2024-03-30T06:53:24.794 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T06:53:24.794 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630 --verbose --name yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/orig.config.yaml 2024-03-30T06:53:24.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T06:53:24.930 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630 branch: wip-yuri11-testing-2024-03-28-0753-reef description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7629630' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri11-testing-2024-03-28-0753-reef ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri11-testing-2024-03-28-0753-reef sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4127 sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri11-testing-2024-03-28-0753-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCp1NYDh5mND57I3ETURfiQ7fqoDP3o6d3ZlghAYPMrkU0/MSYTdfBhjOBfSrbPKToN6CgqW5VCkIWDPzd1CX38= smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMfXVMcbopoc/81qCl9RxIlD5evhICEsqNOexv19l/QmBJ4aBrFJZwxWdExXnqpSLRdVORGZlHFF2f8UAo1Zr0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_19:10:12 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T06:53:24.931 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa; will attempt to use it 2024-03-30T06:53:24.931 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks 2024-03-30T06:53:24.931 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T06:53:24.933 INFO:teuthology.task.internal:Checking packages... 2024-03-30T06:53:24.961 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83' 2024-03-30T06:53:24.961 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T06:53:24.961 INFO:teuthology.packaging:ref: None 2024-03-30T06:53:24.961 INFO:teuthology.packaging:tag: None 2024-03-30T06:53:24.961 INFO:teuthology.packaging:branch: wip-yuri11-testing-2024-03-28-0753-reef 2024-03-30T06:53:24.962 INFO:teuthology.packaging:sha1: 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T06:53:24.962 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri11-testing-2024-03-28-0753-reef 2024-03-30T06:53:25.186 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1055.g55c4b44e 2024-03-30T06:53:25.187 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T06:53:25.194 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T06:53:25.194 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T06:53:25.199 INFO:teuthology.task.internal:Saving configuration 2024-03-30T06:53:25.215 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T06:53:25.220 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T06:53:25.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 06:48:31.693517', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCp1NYDh5mND57I3ETURfiQ7fqoDP3o6d3ZlghAYPMrkU0/MSYTdfBhjOBfSrbPKToN6CgqW5VCkIWDPzd1CX38='} 2024-03-30T06:53:25.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 06:48:31.692125', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMfXVMcbopoc/81qCl9RxIlD5evhICEsqNOexv19l/QmBJ4aBrFJZwxWdExXnqpSLRdVORGZlHFF2f8UAo1Zr0='} 2024-03-30T06:53:25.270 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T06:53:25.277 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-30T06:53:25.277 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-30T06:53:25.277 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T06:53:25.361 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff3b4429f70>, signals=[15]) 2024-03-30T06:53:25.361 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T06:53:25.368 INFO:teuthology.task.internal:Opening connections... 2024-03-30T06:53:25.368 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T06:53:25.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:53:25.483 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T06:53:25.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:53:25.593 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T06:53:25.600 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-03-30T06:53:25.619 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-03-30T06:53:25.620 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="9" 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="9" 2024-03-30T06:53:25.678 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el9" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:LOGO="fedora-logo-icon" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T06:53:25.679 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T06:53:25.680 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-03-30T06:53:25.706 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-03-30T06:53:25.723 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-03-30T06:53:25.724 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-03-30T06:53:25.781 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-03-30T06:53:25.782 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T06:53:25.782 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-03-30T06:53:25.782 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T06:53:25.782 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T06:53:25.782 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T06:53:25.782 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-03-30T06:53:25.802 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T06:53:25.812 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T06:53:25.818 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T06:53:25.818 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-03-30T06:53:25.821 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-03-30T06:53:25.839 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T06:53:25.846 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T06:53:25.846 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-03-30T06:53:25.880 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-03-30T06:53:25.914 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T06:53:25.971 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T06:53:25.987 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T06:53:25.987 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T06:53:25.988 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T06:53:25.989 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T06:53:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T06:53:26.006 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-30T06:53:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T06:53:26.007 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-30T06:53:26.063 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-361.el9.x86_64 2024-03-30T06:53:26.064 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.14.0-361.el9.x86_64 2024-03-30T06:53:26.064 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum install -y kernel 2024-03-30T06:53:26.065 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2024-03-30T06:53:26.065 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2024-03-30T06:53:26.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-03-30T06:53:28.246 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-30T06:53:28.364 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-30T06:53:29.671 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-30T06:53:29.791 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-30T06:53:38.744 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 945 kB/s | 8.0 MB 00:08 2024-03-30T06:53:38.854 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 944 kB/s | 8.0 MB 00:08 2024-03-30T06:53:41.172 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-30T06:53:45.570 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-30T06:53:46.980 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 16 kB 00:00 2024-03-30T06:53:47.246 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-03-30T06:53:48.484 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-30T06:53:51.672 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.1 MB 00:00 2024-03-30T06:53:53.306 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-30T06:53:53.665 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-30T06:53:54.743 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-30T06:53:55.980 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T06:53:55.980 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T06:53:56.089 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T06:53:56.097 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:53:56.098 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 76 M 2024-03-30T06:53:56.099 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-03-30T06:53:56.099 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T06:53:56.258 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-30T06:53:56.967 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-30T06:53:57.642 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-30T06:53:57.942 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-30T06:53:57.945 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T06:53:57.945 INFO:teuthology.orchestra.run.smithi052.stdout:Total 41 MB/s | 76 MB 00:01 2024-03-30T06:53:57.945 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T06:53:58.393 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T06:53:58.393 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T06:53:59.249 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T06:53:59.249 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T06:53:59.840 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-03-30T06:54:01.682 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T06:54:02.298 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T06:54:02.331 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T06:54:02.331 INFO:teuthology.orchestra.run.smithi047.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T06:54:02.378 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:02.431 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T06:54:02.440 INFO:teuthology.orchestra.run.smithi047.stdout:Install 4 Packages 2024-03-30T06:54:02.441 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:02.441 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 76 M 2024-03-30T06:54:02.441 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 123 M 2024-03-30T06:54:02.441 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T06:54:02.592 INFO:teuthology.orchestra.run.smithi047.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-30T06:54:03.167 INFO:teuthology.orchestra.run.smithi047.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-30T06:54:03.667 INFO:teuthology.orchestra.run.smithi047.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-30T06:54:04.101 INFO:teuthology.orchestra.run.smithi047.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-30T06:54:04.103 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T06:54:04.104 INFO:teuthology.orchestra.run.smithi047.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-30T06:54:04.104 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T06:54:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T06:54:04.570 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T06:54:05.295 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:05.430 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:05.436 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T06:54:05.436 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T06:54:07.860 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T06:54:08.466 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T06:54:08.553 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:10.507 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:10.545 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:11.433 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:11.562 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:16.676 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:16.728 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:21.865 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:46.561 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:46.561 INFO:teuthology.orchestra.run.smithi052.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T06:54:46.561 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:54:46.606 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T06:54:46.933 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:46.934 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:47.333 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:47.333 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:54:47.333 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:54:47.334 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T06:54:47.683 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T06:54:47.782 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T06:54:47.783 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T06:54:47.783 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T06:54:47.783 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T06:54:47.783 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T06:54:47.783 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T06:54:47.784 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T06:54:47.866 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:47.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T06:54:47.869 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T06:54:47.908 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T06:54:47.909 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T06:54:47.910 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T06:54:47.910 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-434.el9.x86_64 2024-03-30T06:54:47.911 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T06:54:48.488 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-03-30T06:54:49.896 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T06:54:49.924 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-03-30T06:54:49.925 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-30T06:54:49.941 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.f0lDxhRuaK 2024-03-30T06:54:49.941 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.f0lDxhRuaK 2024-03-30T06:54:50.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.f0lDxhRuaK 2024-03-30T06:54:50.153 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.f0lDxhRuaK is 6KB 2024-03-30T06:54:50.202 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.f0lDxhRuaK 2024-03-30T06:54:50.217 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T06:54:50.285 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T06:54:50.286 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T06:54:50.286 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T06:54:50.286 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T06:54:50.287 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T06:54:50.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-03-30T06:54:52.867 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:52.867 INFO:teuthology.orchestra.run.smithi047.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T06:54:52.867 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:52.912 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:53.298 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:53.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T06:54:53.298 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T06:54:53.299 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T06:54:53.724 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T06:54:53.724 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:53.725 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T06:54:54.036 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T06:54:54.152 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:54.153 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T06:54:54.153 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T06:54:54.153 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T06:54:54.153 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T06:54:54.153 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T06:54:54.153 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T06:54:54.153 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T06:54:54.238 INFO:teuthology.orchestra.run.smithi047.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T06:54:54.238 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T06:54:54.239 DEBUG:teuthology.orchestra.run.smithi047:> sudo rpm -qi grub2-tools 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Name : grub2-tools 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Epoch : 1 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Version : 2.06 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Release : 68.el9 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: x86_64 2024-03-30T06:54:54.277 INFO:teuthology.orchestra.run.smithi047.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:Group : Unspecified 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:Size : 8264254 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:License : GPLv3+ 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T06:54:54.278 INFO:teuthology.orchestra.run.smithi047.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:Packager : builder@centos.org 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:Vendor : CentOS 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:Summary : Support tools for GRUB. 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout:Description : 2024-03-30T06:54:54.279 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout:hardware devices. 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T06:54:54.280 INFO:teuthology.orchestra.run.smithi047.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T06:54:54.282 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T06:54:54.282 INFO:teuthology.task.kernel:Updating grub on smithi047 to boot 5.14.0-434.el9.x86_64 2024-03-30T06:54:54.282 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T06:54:54.945 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-03-30T06:54:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T06:54:56.393 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-03-30T06:54:56.394 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-30T06:54:56.450 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.0pQZufVQWW 2024-03-30T06:54:56.451 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0pQZufVQWW 2024-03-30T06:54:56.523 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.0pQZufVQWW 2024-03-30T06:54:56.659 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.0pQZufVQWW is 6KB 2024-03-30T06:54:56.706 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.0pQZufVQWW 2024-03-30T06:54:56.720 DEBUG:teuthology.orchestra.run.smithi047:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T06:54:56.798 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T06:54:56.798 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T06:54:56.798 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T06:54:56.798 INFO:teuthology.orchestra.run.smithi047.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T06:54:56.800 DEBUG:teuthology.orchestra.run.smithi047:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T06:54:56.952 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-03-30T06:55:20.464 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T06:55:20.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-30T06:55:20.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:55:26.958 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T06:55:26.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T06:55:26.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:55:45.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-03-30T06:55:54.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T06:55:54.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:56:20.491 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T06:56:29.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-30T06:56:29.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:56:29.901 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-30T06:56:30.323 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-30T06:56:30.324 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-30T06:56:30.324 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-30T06:56:30.337 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-434.el9.x86_64 2024-03-30T06:56:30.337 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T06:56:30.337 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T06:56:30.338 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T06:56:31.338 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T06:56:31.338 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T06:56:31.428 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-03-30T06:56:31.465 DEBUG:teuthology.parallel:result is None 2024-03-30T06:56:54.450 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T06:57:06.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T06:57:06.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T06:57:06.748 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-30T06:57:07.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T06:57:07.169 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T06:57:07.169 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-03-30T06:57:07.185 INFO:teuthology.orchestra.run.smithi047.stdout:5.14.0-434.el9.x86_64 2024-03-30T06:57:07.185 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T06:57:07.185 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T06:57:07.185 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T06:57:08.186 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T06:57:08.187 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T06:57:08.251 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-03-30T06:57:08.286 DEBUG:teuthology.parallel:result is None 2024-03-30T06:57:08.289 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T06:57:08.300 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T06:57:08.300 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T06:57:08.303 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T06:57:08.321 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T06:57:08.328 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T06:57:08.336 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T06:57:08.336 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T06:57:08.360 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T06:57:08.389 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T06:57:08.395 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T06:57:08.395 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T06:57:08.423 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T06:57:08.459 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T06:57:08.465 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T06:57:08.473 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T06:57:08.479 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T06:57:08.481 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T06:57:08.489 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T06:57:08.489 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T06:57:08.518 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T06:57:08.551 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T06:57:08.560 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T06:57:08.560 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T06:57:08.588 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T06:57:08.606 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T06:57:08.683 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T06:57:08.731 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T06:57:08.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T06:57:08.857 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T06:57:08.857 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T06:57:08.923 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T06:57:08.971 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T06:57:09.016 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T06:57:09.099 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T06:57:09.143 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T06:57:09.143 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T06:57:09.211 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-03-30T06:57:09.213 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-03-30T06:57:09.260 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T06:57:09.315 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T06:57:09.669 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T06:57:09.677 INFO:teuthology.task.internal:Starting timer... 2024-03-30T06:57:09.677 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T06:57:09.687 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T06:57:09.695 DEBUG:teuthology.orchestra.run.smithi047:> sudo service auditd rotate 2024-03-30T06:57:09.758 INFO:teuthology.orchestra.run.smithi047.stdout:Rotating logs: 2024-03-30T06:57:09.760 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-03-30T06:57:09.825 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-03-30T06:57:09.827 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T06:57:09.828 DEBUG:teuthology.orchestra.run.smithi047:> /usr/sbin/getenforce 2024-03-30T06:57:09.848 INFO:teuthology.orchestra.run.smithi047.stdout:Permissive 2024-03-30T06:57:09.849 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-03-30T06:57:09.899 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-03-30T06:57:09.900 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi047.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi052.front.sepia.ceph.com': 'permissive'} 2024-03-30T06:57:09.900 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T06:57:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T06:57:09.927 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T06:57:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T06:57:09.955 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T06:57:09.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo /usr/sbin/setenforce permissive 2024-03-30T06:57:10.000 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-03-30T06:57:10.043 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T06:57:10.052 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-30T06:57:10.058 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-30T06:57:10.187 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T06:57:10.208 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T06:57:10.209 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T07:02:24.792 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2024-03-30T07:02:24.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T07:02:24.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T07:02:24.890 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-03-30T07:02:24.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-03-30T07:02:24.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-30T07:02:24.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T07:02:25.058 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-30T07:02:25.138 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-30T07:02:25.139 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T07:02:25.149 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T07:02:25.150 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T07:02:25.150 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T07:02:25.153 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T07:02:25.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T07:02:25.183 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T07:02:25.201 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T07:02:25.209 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T07:02:25.226 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T07:02:25.231 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: ntpd: command not found 2024-03-30T07:02:25.245 INFO:teuthology.orchestra.run.smithi047.stdout:506 Cannot talk to daemon 2024-03-30T07:02:25.256 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-03-30T07:02:25.261 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T07:02:25.270 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-03-30T07:02:25.278 INFO:teuthology.orchestra.run.smithi047.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T07:02:25.286 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T07:02:25.302 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T07:02:25.329 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-30T07:02:25.332 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T07:02:25.332 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-30T07:02:25.333 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.333 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.333 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.333 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.355 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.359 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:02:25.359 INFO:teuthology.run_tasks:Running task install... 2024-03-30T07:02:25.368 DEBUG:teuthology.task.install:project ceph 2024-03-30T07:02:25.368 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83'}} 2024-03-30T07:02:25.368 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83'} 2024-03-30T07:02:25.368 INFO:teuthology.task.install:Using flavor: default 2024-03-30T07:02:25.374 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-30T07:02:25.375 INFO:teuthology.task.install:extra packages: [] 2024-03-30T07:02:25.375 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83', 'tag': None, 'wait_for_package': False} 2024-03-30T07:02:25.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:02:25.378 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83', 'tag': None, 'wait_for_package': False} 2024-03-30T07:02:25.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:02:25.594 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/ 2024-03-30T07:02:25.594 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1055.g55c4b44e 2024-03-30T07:02:25.601 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/ 2024-03-30T07:02:25.601 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1055.g55c4b44e 2024-03-30T07:02:25.705 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T07:02:25.705 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:02:25.705 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T07:02:25.734 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-30T07:02:25.735 DEBUG:teuthology.orchestra.run.smithi047:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T07:02:25.768 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-28-0753-reef/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T07:02:25.768 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:02:25.768 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T07:02:25.797 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-30T07:02:25.798 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T07:02:25.803 DEBUG:teuthology.orchestra.run.smithi047:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T07:02:25.866 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T07:02:25.883 DEBUG:teuthology.orchestra.run.smithi047:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T07:02:25.947 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T07:02:25.949 INFO:teuthology.orchestra.run.smithi047.stdout:check_obsoletes = 1 2024-03-30T07:02:25.950 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum clean all 2024-03-30T07:02:26.014 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-03-30T07:02:26.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-03-30T07:02:26.198 INFO:teuthology.orchestra.run.smithi047.stdout:74 files removed 2024-03-30T07:02:26.225 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw 2024-03-30T07:02:26.266 INFO:teuthology.orchestra.run.smithi052.stdout:74 files removed 2024-03-30T07:02:26.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-03-30T07:02:26.884 INFO:teuthology.orchestra.run.smithi047.stdout:ceph packages for x86_64 218 kB/s | 77 kB 00:00 2024-03-30T07:02:26.927 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 235 kB/s | 77 kB 00:00 2024-03-30T07:02:27.318 INFO:teuthology.orchestra.run.smithi047.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-03-30T07:02:27.328 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-03-30T07:02:27.696 INFO:teuthology.orchestra.run.smithi047.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-30T07:02:27.745 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-30T07:02:28.214 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-30T07:02:28.221 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-30T07:02:29.748 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-30T07:02:29.754 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-30T07:02:30.637 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-30T07:02:32.097 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:01 2024-03-30T07:02:34.863 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-03-30T07:02:38.947 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-30T07:02:41.282 INFO:teuthology.orchestra.run.smithi047.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-30T07:02:43.283 INFO:teuthology.orchestra.run.smithi047.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-30T07:02:48.491 INFO:teuthology.orchestra.run.smithi047.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-30T07:02:51.170 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:02:51.189 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:02:51.189 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:02:51.189 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 7.7 M 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrading: 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.3 M 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.0 M 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.1 M 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 18 M 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 25 k 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-30T07:02:51.190 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 709 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 474 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 4.5 M 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse 2024-03-30T07:02:51.191 INFO:teuthology.orchestra.run.smithi047.stdout: x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 45 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 129 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 162 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 320 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 298 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 99 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout:Install 28 Packages 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout:Upgrade 2 Packages 2024-03-30T07:02:51.192 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:02:51.193 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 53 M 2024-03-30T07:02:51.193 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:02:51.867 INFO:teuthology.orchestra.run.smithi047.stdout:(1/30): ceph-base-18.2.2-1055.g55c4b44e.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-30T07:02:51.901 INFO:teuthology.orchestra.run.smithi047.stdout:(2/30): ceph-selinux-18.2.2-1055.g55c4b44e.el9. 759 kB/s | 25 kB 00:00 2024-03-30T07:02:51.968 INFO:teuthology.orchestra.run.smithi047.stdout:(3/30): libcephfs2-18.2.2-1055.g55c4b44e.el9.x8 10 MB/s | 709 kB 00:00 2024-03-30T07:02:52.027 INFO:teuthology.orchestra.run.smithi047.stdout:(4/30): libradosstriper1-18.2.2-1055.g55c4b44e. 8.0 MB/s | 474 kB 00:00 2024-03-30T07:02:52.294 INFO:teuthology.orchestra.run.smithi047.stdout:(5/30): librgw2-18.2.2-1055.g55c4b44e.el9.x86_6 17 MB/s | 4.5 MB 00:00 2024-03-30T07:02:52.327 INFO:teuthology.orchestra.run.smithi047.stdout:(6/30): python3-ceph-argparse-18.2.2-1055.g55c4 1.3 MB/s | 45 kB 00:00 2024-03-30T07:02:52.377 INFO:teuthology.orchestra.run.smithi047.stdout:(7/30): python3-ceph-common-18.2.2-1055.g55c4b4 2.5 MB/s | 129 kB 00:00 2024-03-30T07:02:52.444 INFO:teuthology.orchestra.run.smithi047.stdout:(8/30): python3-cephfs-18.2.2-1055.g55c4b44e.el 2.4 MB/s | 162 kB 00:00 2024-03-30T07:02:52.503 INFO:teuthology.orchestra.run.smithi047.stdout:(9/30): python3-rados-18.2.2-1055.g55c4b44e.el9 5.4 MB/s | 320 kB 00:00 2024-03-30T07:02:52.578 INFO:teuthology.orchestra.run.smithi047.stdout:(10/30): python3-rbd-18.2.2-1055.g55c4b44e.el9. 3.9 MB/s | 298 kB 00:00 2024-03-30T07:02:52.637 INFO:teuthology.orchestra.run.smithi047.stdout:(11/30): python3-rgw-18.2.2-1055.g55c4b44e.el9. 1.7 MB/s | 99 kB 00:00 2024-03-30T07:02:52.737 INFO:teuthology.orchestra.run.smithi047.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-30T07:02:52.779 INFO:teuthology.orchestra.run.smithi047.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-30T07:02:52.804 INFO:teuthology.orchestra.run.smithi047.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-30T07:02:52.871 INFO:teuthology.orchestra.run.smithi047.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-30T07:02:52.913 INFO:teuthology.orchestra.run.smithi047.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-30T07:02:52.947 INFO:teuthology.orchestra.run.smithi047.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-30T07:02:53.047 INFO:teuthology.orchestra.run.smithi047.stdout:(18/30): ceph-radosgw-18.2.2-1055.g55c4b44e.el9 4.6 MB/s | 7.7 MB 00:01 2024-03-30T07:02:53.214 INFO:teuthology.orchestra.run.smithi047.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 415 kB/s | 111 kB 00:00 2024-03-30T07:02:53.289 INFO:teuthology.orchestra.run.smithi047.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-30T07:02:53.348 INFO:teuthology.orchestra.run.smithi047.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-30T07:02:53.406 INFO:teuthology.orchestra.run.smithi047.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-03-30T07:02:53.582 INFO:teuthology.orchestra.run.smithi047.stdout:(23/30): ceph-common-18.2.2-1055.g55c4b44e.el9. 8.3 MB/s | 18 MB 00:02 2024-03-30T07:02:53.607 INFO:teuthology.orchestra.run.smithi047.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-03-30T07:02:53.699 INFO:teuthology.orchestra.run.smithi047.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-30T07:02:53.766 INFO:teuthology.orchestra.run.smithi047.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-03-30T07:02:53.958 INFO:teuthology.orchestra.run.smithi047.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-30T07:02:54.025 INFO:teuthology.orchestra.run.smithi047.stdout:(28/30): librados2-18.2.2-1055.g55c4b44e.el9.x8 13 MB/s | 3.3 MB 00:00 2024-03-30T07:02:54.083 INFO:teuthology.orchestra.run.smithi047.stdout:(29/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-03-30T07:02:54.234 INFO:teuthology.orchestra.run.smithi047.stdout:(30/30): librbd1-18.2.2-1055.g55c4b44e.el9.x86_ 11 MB/s | 3.0 MB 00:00 2024-03-30T07:02:54.237 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:02:54.237 INFO:teuthology.orchestra.run.smithi047.stdout:Total 17 MB/s | 53 MB 00:03 2024-03-30T07:02:54.237 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:02:54.275 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:02:54.275 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:02:54.737 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:02:54.737 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:02:56.166 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:02:56.259 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-30T07:02:56.288 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-30T07:02:56.339 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-30T07:02:56.732 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-30T07:02:56.762 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:02:56.846 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:02:56.933 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/32 2024-03-30T07:02:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:02:57.027 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:02:57.060 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-30T07:02:57.109 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9. 9/32 2024-03-30T07:02:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 10/32 2024-03-30T07:02:57.199 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 11/32 2024-03-30T07:02:57.265 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 11/32 2024-03-30T07:02:57.375 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-30T07:02:57.411 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-30T07:02:57.448 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-30T07:02:57.480 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-30T07:02:57.506 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-30T07:02:57.542 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-30T07:02:57.575 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-30T07:02:57.895 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-30T07:02:57.921 INFO:teuthology.orchestra.run.smithi047.stdout: Upgrading : librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 20/32 2024-03-30T07:02:58.020 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 20/32 2024-03-30T07:02:58.054 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 21/32 2024-03-30T07:02:58.103 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-30T07:02:58.532 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-30T07:02:58.558 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 24/32 2024-03-30T07:02:58.628 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 24/32 2024-03-30T07:02:58.700 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 25/32 2024-03-30T07:02:58.726 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x8 26/32 2024-03-30T07:03:00.908 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:03:00.939 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:03:01.491 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:03:01.533 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 28/32 2024-03-30T07:03:01.606 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 28/32 2024-03-30T07:03:01.606 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T07:03:01.606 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T07:03:01.606 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:01.643 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /sys 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /proc 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /mnt 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /var/tmp 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /home 2024-03-30T07:03:11.798 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /root 2024-03-30T07:03:11.799 INFO:teuthology.orchestra.run.smithi047.stdout:skipping the directory /tmp 2024-03-30T07:03:11.799 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:12.567 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/32 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/32 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T07:03:12.592 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:12.595 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-30T07:03:12.671 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-30T07:03:12.671 INFO:teuthology.orchestra.run.smithi047.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 3/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 4/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 6/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:03:13.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9. 8/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x8 9/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 10/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 11/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 12/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 13/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-30T07:03:13.737 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-30T07:03:13.738 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/32 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout:Upgraded: 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.806 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.807 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:14.808 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:15.149 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test 2024-03-30T07:03:15.741 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:27 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:15.897 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:15.898 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:15.898 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:03:15.898 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:15.898 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:15.898 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 40 M 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 166 k 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-30T07:03:15.899 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:15.900 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 41 M 2024-03-30T07:03:15.900 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 164 M 2024-03-30T07:03:15.900 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:16.210 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-30T07:03:16.252 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-30T07:03:16.277 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-30T07:03:16.419 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): libcephsqlite-18.2.2-1055.g55c4b44e.el9. 431 kB/s | 166 kB 00:00 2024-03-30T07:03:16.536 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 723 kB/s | 187 kB 00:00 2024-03-30T07:03:18.428 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): ceph-test-18.2.2-1055.g55c4b44e.el9.x86_ 17 MB/s | 40 MB 00:02 2024-03-30T07:03:18.431 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:18.431 INFO:teuthology.orchestra.run.smithi047.stdout:Total 16 MB/s | 41 MB 00:02 2024-03-30T07:03:18.445 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:18.467 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:18.467 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:18.817 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:18.817 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:19.413 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:19.496 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T07:03:19.530 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T07:03:19.564 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T07:03:19.604 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T07:03:19.631 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/6 2024-03-30T07:03:23.556 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/6 2024-03-30T07:03:23.640 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/6 2024-03-30T07:03:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/6 2024-03-30T07:03:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/6 2024-03-30T07:03:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/6 2024-03-30T07:03:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T07:03:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T07:03:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: jq-1.6-16.el9.x86_64 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:26.069 INFO:teuthology.orchestra.run.smithi047.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T07:03:26.070 INFO:teuthology.orchestra.run.smithi047.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T07:03:26.070 INFO:teuthology.orchestra.run.smithi047.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T07:03:26.070 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:26.070 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:26.346 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph 2024-03-30T07:03:26.918 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:38 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:27.077 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: ceph x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 6.5 k 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 2.1 M 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.5 M 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 246 k 2024-03-30T07:03:27.080 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 4.7 M 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 17 M 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T07:03:27.081 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T07:03:27.082 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout:========================================================================================= 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout:Install 33 Packages 2024-03-30T07:03:27.083 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:27.085 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 29 M 2024-03-30T07:03:27.085 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 104 M 2024-03-30T07:03:27.085 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:27.439 INFO:teuthology.orchestra.run.smithi047.stdout:(1/33): ceph-18.2.2-1055.g55c4b44e.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2024-03-30T07:03:27.698 INFO:teuthology.orchestra.run.smithi047.stdout:(2/33): ceph-mgr-18.2.2-1055.g55c4b44e.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-03-30T07:03:27.739 INFO:teuthology.orchestra.run.smithi047.stdout:(3/33): ceph-mds-18.2.2-1055.g55c4b44e.el9.x86_ 4.5 MB/s | 2.1 MB 00:00 2024-03-30T07:03:27.831 INFO:teuthology.orchestra.run.smithi047.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1055.g55c4 2.6 MB/s | 246 kB 00:00 2024-03-30T07:03:27.940 INFO:teuthology.orchestra.run.smithi047.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-30T07:03:27.973 INFO:teuthology.orchestra.run.smithi047.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-30T07:03:28.007 INFO:teuthology.orchestra.run.smithi047.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-30T07:03:28.041 INFO:teuthology.orchestra.run.smithi047.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-30T07:03:28.091 INFO:teuthology.orchestra.run.smithi047.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-30T07:03:28.225 INFO:teuthology.orchestra.run.smithi047.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-30T07:03:28.267 INFO:teuthology.orchestra.run.smithi047.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-30T07:03:28.309 INFO:teuthology.orchestra.run.smithi047.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-30T07:03:28.350 INFO:teuthology.orchestra.run.smithi047.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-30T07:03:28.426 INFO:teuthology.orchestra.run.smithi047.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-30T07:03:28.585 INFO:teuthology.orchestra.run.smithi047.stdout:(15/33): ceph-osd-18.2.2-1055.g55c4b44e.el9.x86 19 MB/s | 17 MB 00:00 2024-03-30T07:03:28.660 INFO:teuthology.orchestra.run.smithi047.stdout:(16/33): ceph-mon-18.2.2-1055.g55c4b44e.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-03-30T07:03:28.752 INFO:teuthology.orchestra.run.smithi047.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 133 kB/s | 43 kB 00:00 2024-03-30T07:03:28.827 INFO:teuthology.orchestra.run.smithi047.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-30T07:03:28.911 INFO:teuthology.orchestra.run.smithi047.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-30T07:03:28.953 INFO:teuthology.orchestra.run.smithi047.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2024-03-30T07:03:29.028 INFO:teuthology.orchestra.run.smithi047.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-30T07:03:29.053 INFO:teuthology.orchestra.run.smithi047.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-30T07:03:29.104 INFO:teuthology.orchestra.run.smithi047.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-30T07:03:29.129 INFO:teuthology.orchestra.run.smithi047.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-03-30T07:03:29.279 INFO:teuthology.orchestra.run.smithi047.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 450 kB/s | 79 kB 00:00 2024-03-30T07:03:29.312 INFO:teuthology.orchestra.run.smithi047.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 549 kB/s | 358 kB 00:00 2024-03-30T07:03:29.338 INFO:teuthology.orchestra.run.smithi047.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-30T07:03:29.363 INFO:teuthology.orchestra.run.smithi047.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-03-30T07:03:29.438 INFO:teuthology.orchestra.run.smithi047.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-30T07:03:29.463 INFO:teuthology.orchestra.run.smithi047.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 597 kB/s | 90 kB 00:00 2024-03-30T07:03:29.564 INFO:teuthology.orchestra.run.smithi047.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-30T07:03:29.614 INFO:teuthology.orchestra.run.smithi047.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 916 kB/s | 230 kB 00:00 2024-03-30T07:03:29.706 INFO:teuthology.orchestra.run.smithi047.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-30T07:03:29.709 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:29.710 INFO:teuthology.orchestra.run.smithi047.stdout:Total 11 MB/s | 29 MB 00:02 2024-03-30T07:03:29.724 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:29.768 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:29.768 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:30.146 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:30.146 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:31.420 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:31.476 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-30T07:03:31.508 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-30T07:03:31.541 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-30T07:03:31.637 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-30T07:03:31.714 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-30T07:03:31.761 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-30T07:03:31.845 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-30T07:03:31.896 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-30T07:03:31.933 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-30T07:03:31.992 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-30T07:03:32.056 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-30T07:03:32.260 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-30T07:03:32.312 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-30T07:03:32.355 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-30T07:03:32.394 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-30T07:03:32.424 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-30T07:03:32.502 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-30T07:03:32.534 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-30T07:03:32.567 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-30T07:03:32.608 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-30T07:03:32.652 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-30T07:03:32.685 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-30T07:03:32.781 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-30T07:03:32.817 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-30T07:03:32.893 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-30T07:03:33.044 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-30T07:03:33.183 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-30T07:03:33.359 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9. 28/33 2024-03-30T07:03:33.388 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/33 2024-03-30T07:03:33.412 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/33 2024-03-30T07:03:33.412 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:33.412 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T07:03:33.413 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T07:03:33.413 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T07:03:33.413 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:34.989 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/33 2024-03-30T07:03:35.017 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/33 2024-03-30T07:03:35.017 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:35.017 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T07:03:35.017 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T07:03:35.018 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T07:03:35.018 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:35.481 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/33 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/33 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T07:03:35.505 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:35.716 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 32/33 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 32/33 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T07:03:35.739 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:35.807 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 33/33 2024-03-30T07:03:37.517 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 33/33 2024-03-30T07:03:37.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/33 2024-03-30T07:03:37.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/33 2024-03-30T07:03:37.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 3/33 2024-03-30T07:03:37.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 4/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9. 6/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-30T07:03:37.518 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-30T07:03:37.519 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T07:03:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T07:03:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T07:03:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T07:03:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T07:03:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T07:03:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:38.580 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:38.907 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base 2024-03-30T07:03:39.489 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:51 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:39.591 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:03:39.645 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:39.646 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:03:39.646 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:39.718 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install cephadm 2024-03-30T07:03:40.285 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:52 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:40.441 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 224 k 2024-03-30T07:03:40.442 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 224 k 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 216 k 2024-03-30T07:03:40.443 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:40.681 INFO:teuthology.orchestra.run.smithi047.stdout:cephadm-18.2.2-1055.g55c4b44e.el9.noarch.rpm 944 kB/s | 224 kB 00:00 2024-03-30T07:03:40.681 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:40.681 INFO:teuthology.orchestra.run.smithi047.stdout:Total 938 kB/s | 224 kB 00:00 2024-03-30T07:03:40.682 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:40.687 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:40.687 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:40.690 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:40.690 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:40.762 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:41.167 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:03:41.263 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:03:41.615 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:03:41.891 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:03:41.891 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:41.892 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:41.892 INFO:teuthology.orchestra.run.smithi047.stdout: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:03:41.892 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:41.892 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:42.110 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T07:03:42.683 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:54 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:42.841 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 143 k 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:42.842 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:03:42.843 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:42.843 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 143 k 2024-03-30T07:03:42.843 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 395 k 2024-03-30T07:03:42.843 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:43.446 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-18.2.2-1055.g55c4b4 236 kB/s | 143 kB 00:00 2024-03-30T07:03:43.447 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:43.447 INFO:teuthology.orchestra.run.smithi047.stdout:Total 236 kB/s | 143 kB 00:00 2024-03-30T07:03:43.447 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:43.456 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:43.456 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:43.511 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:43.511 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:43.778 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:43.839 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:03:43.863 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:03:43.863 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:03:43.863 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T07:03:43.863 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:44.848 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:03:44.849 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:44.849 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:44.849 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:03:44.849 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:44.849 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:45.118 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr 2024-03-30T07:03:45.692 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:57 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:45.794 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:03:45.849 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:45.850 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:03:45.850 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:45.913 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T07:03:46.483 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:00:58 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:46.641 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:46.641 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:46.641 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:03:46.641 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 3.5 M 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 23 k 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 15 k 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout:Install 5 Packages 2024-03-30T07:03:46.642 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:46.643 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.8 M 2024-03-30T07:03:46.643 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 87 M 2024-03-30T07:03:46.643 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:47.251 INFO:teuthology.orchestra.run.smithi047.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1055.g55c 51 kB/s | 23 kB 00:00 2024-03-30T07:03:47.285 INFO:teuthology.orchestra.run.smithi047.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1055.g55c4 30 kB/s | 15 kB 00:00 2024-03-30T07:03:47.718 INFO:teuthology.orchestra.run.smithi047.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 66 kB/s | 31 kB 00:00 2024-03-30T07:03:47.893 INFO:teuthology.orchestra.run.smithi047.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 310 kB/s | 188 kB 00:00 2024-03-30T07:03:48.260 INFO:teuthology.orchestra.run.smithi047.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1055.g55c4b44e 2.4 MB/s | 3.5 MB 00:01 2024-03-30T07:03:48.264 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:48.264 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.3 MB/s | 3.8 MB 00:01 2024-03-30T07:03:48.265 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:48.274 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:48.274 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:48.358 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:48.358 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:48.609 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:48.700 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-30T07:03:48.732 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-30T07:03:48.761 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.n 3/5 2024-03-30T07:03:49.867 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9. 4/5 2024-03-30T07:03:49.897 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 5/5 2024-03-30T07:03:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 5/5 2024-03-30T07:03:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9. 1/5 2024-03-30T07:03:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 2/5 2024-03-30T07:03:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.n 3/5 2024-03-30T07:03:50.997 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-30T07:03:51.284 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-30T07:03:51.284 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:51.284 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:03:51.284 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:51.285 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:03:51.496 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T07:03:52.067 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:04 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:03:52.226 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 7.4 M 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T07:03:52.227 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================================ 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout:Install 12 Packages 2024-03-30T07:03:52.228 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:03:52.229 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 37 M 2024-03-30T07:03:52.229 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 209 M 2024-03-30T07:03:52.229 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:03:52.918 INFO:teuthology.orchestra.run.smithi047.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2024-03-30T07:03:52.951 INFO:teuthology.orchestra.run.smithi047.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-30T07:03:52.993 INFO:teuthology.orchestra.run.smithi047.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-03-30T07:03:53.051 INFO:teuthology.orchestra.run.smithi047.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-30T07:03:53.219 INFO:teuthology.orchestra.run.smithi047.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-30T07:03:53.285 INFO:teuthology.orchestra.run.smithi047.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-30T07:03:53.311 INFO:teuthology.orchestra.run.smithi047.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-30T07:03:53.553 INFO:teuthology.orchestra.run.smithi047.stdout:(8/12): ceph-mgr-diskprediction-local-18.2.2-10 9.5 MB/s | 7.4 MB 00:00 2024-03-30T07:03:53.703 INFO:teuthology.orchestra.run.smithi047.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-30T07:03:53.820 INFO:teuthology.orchestra.run.smithi047.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 732 kB/s | 195 kB 00:00 2024-03-30T07:03:53.854 INFO:teuthology.orchestra.run.smithi047.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-30T07:03:53.879 INFO:teuthology.orchestra.run.smithi047.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-30T07:03:53.881 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:03:53.881 INFO:teuthology.orchestra.run.smithi047.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-30T07:03:53.922 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:03:53.946 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:03:53.946 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:03:54.386 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:03:54.386 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:03:55.220 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:03:55.301 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T07:03:55.341 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T07:03:55.438 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T07:03:55.486 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T07:03:55.676 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T07:03:55.710 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T07:03:55.789 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T07:03:56.467 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T07:03:56.651 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T07:03:57.985 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T07:03:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T07:03:59.101 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 12/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 12/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 1/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T07:04:01.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T07:04:02.017 INFO:teuthology.orchestra.run.smithi047.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T07:04:02.018 INFO:teuthology.orchestra.run.smithi047.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T07:04:02.019 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:02.019 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:02.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-rook 2024-03-30T07:04:02.931 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:14 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:03.092 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 49 k 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T07:04:03.093 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:03.094 INFO:teuthology.orchestra.run.smithi047.stdout:Install 13 Packages 2024-03-30T07:04:03.095 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:03.095 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 2.0 M 2024-03-30T07:04:03.095 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 26 M 2024-03-30T07:04:03.095 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:03.368 INFO:teuthology.orchestra.run.smithi047.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 146 kB/s | 18 kB 00:00 2024-03-30T07:04:03.393 INFO:teuthology.orchestra.run.smithi047.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 164 kB/s | 24 kB 00:00 2024-03-30T07:04:03.568 INFO:teuthology.orchestra.run.smithi047.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 873 kB/s | 174 kB 00:00 2024-03-30T07:04:03.602 INFO:teuthology.orchestra.run.smithi047.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 634 kB/s | 132 kB 00:00 2024-03-30T07:04:03.627 INFO:teuthology.orchestra.run.smithi047.stdout:(5/13): ceph-mgr-rook-18.2.2-1055.g55c4b44e.el9 129 kB/s | 49 kB 00:00 2024-03-30T07:04:03.652 INFO:teuthology.orchestra.run.smithi047.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 211 kB 00:00 2024-03-30T07:04:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-30T07:04:03.728 INFO:teuthology.orchestra.run.smithi047.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-30T07:04:03.762 INFO:teuthology.orchestra.run.smithi047.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-03-30T07:04:03.812 INFO:teuthology.orchestra.run.smithi047.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.5 MB/s | 203 kB 00:00 2024-03-30T07:04:03.837 INFO:teuthology.orchestra.run.smithi047.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-03-30T07:04:03.862 INFO:teuthology.orchestra.run.smithi047.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-30T07:04:03.921 INFO:teuthology.orchestra.run.smithi047.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-03-30T07:04:03.923 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:03.923 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-30T07:04:03.923 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:03.937 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:03.937 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:04.031 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:04.031 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:04.543 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:04.725 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T07:04:04.790 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T07:04:04.846 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T07:04:04.882 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T07:04:04.924 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T07:04:05.020 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T07:04:05.132 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T07:04:05.192 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T07:04:05.959 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T07:04:06.022 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T07:04:06.056 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T07:04:06.104 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T07:04:06.134 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 13/13 2024-03-30T07:04:06.972 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 13/13 2024-03-30T07:04:06.972 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 1/13 2024-03-30T07:04:06.972 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T07:04:06.972 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T07:04:06.973 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T07:04:07.341 INFO:teuthology.orchestra.run.smithi047.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:07.342 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:07.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T07:04:08.154 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:20 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:08.310 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 138 k 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:04:08.311 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout:Install 6 Packages 2024-03-30T07:04:08.312 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:08.313 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 6.8 M 2024-03-30T07:04:08.313 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 33 M 2024-03-30T07:04:08.313 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:08.773 INFO:teuthology.orchestra.run.smithi047.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-30T07:04:08.865 INFO:teuthology.orchestra.run.smithi047.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-30T07:04:08.982 INFO:teuthology.orchestra.run.smithi047.stdout:(3/6): ceph-mgr-cephadm-18.2.2-1055.g55c4b44e.e 341 kB/s | 138 kB 00:00 2024-03-30T07:04:09.366 INFO:teuthology.orchestra.run.smithi047.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-30T07:04:09.424 INFO:teuthology.orchestra.run.smithi047.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-03-30T07:04:09.483 INFO:teuthology.orchestra.run.smithi047.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 775 kB/s | 548 kB 00:00 2024-03-30T07:04:09.487 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:09.487 INFO:teuthology.orchestra.run.smithi047.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-30T07:04:09.500 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:09.511 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:09.511 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:09.603 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:09.603 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:09.914 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:10.086 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T07:04:10.135 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T07:04:10.698 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T07:04:10.782 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T07:04:10.843 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T07:04:10.874 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 6/6 2024-03-30T07:04:11.686 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 6/6 2024-03-30T07:04:11.687 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/6 2024-03-30T07:04:11.687 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T07:04:11.687 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T07:04:11.687 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T07:04:11.687 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:11.989 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:12.224 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse 2024-03-30T07:04:12.798 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:24 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:12.955 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 849 k 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-30T07:04:12.956 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 927 k 2024-03-30T07:04:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.7 M 2024-03-30T07:04:12.957 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:13.228 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 290 kB/s | 79 kB 00:00 2024-03-30T07:04:13.262 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-fuse-18.2.2-1055.g55c4b44e.el9.x86_ 2.7 MB/s | 849 kB 00:00 2024-03-30T07:04:13.263 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:13.263 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.0 MB/s | 927 kB 00:00 2024-03-30T07:04:13.263 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:13.284 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:13.284 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:13.359 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:13.359 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:13.547 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:13.681 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T07:04:13.757 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:04:14.431 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:04:14.431 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/2 2024-03-30T07:04:14.731 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T07:04:14.731 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:14.731 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:14.732 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T07:04:14.732 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:14.732 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:15.009 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-volume 2024-03-30T07:04:15.588 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:27 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:15.746 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repository Size 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 264 k 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:15.747 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:15.748 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-30T07:04:15.748 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:15.748 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 332 k 2024-03-30T07:04:15.748 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.5 M 2024-03-30T07:04:15.748 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:15.884 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 508 kB/s | 69 kB 00:00 2024-03-30T07:04:16.125 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-volume-18.2.2-1055.g55c4b44e.el9.no 700 kB/s | 264 kB 00:00 2024-03-30T07:04:16.126 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:16.127 INFO:teuthology.orchestra.run.smithi047.stdout:Total 876 kB/s | 332 kB 00:00 2024-03-30T07:04:16.127 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:16.136 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:16.136 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:16.174 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:16.174 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:16.329 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:16.499 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T07:04:16.536 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2/2 2024-03-30T07:04:16.552 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2/2 2024-03-30T07:04:16.552 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:04:16.552 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T07:04:16.552 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:17.280 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 1/2 2024-03-30T07:04:17.621 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T07:04:17.621 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:17.622 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:17.622 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:04:17.622 INFO:teuthology.orchestra.run.smithi047.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T07:04:17.622 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:17.622 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:18.001 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel 2024-03-30T07:04:18.578 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:30 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:18.735 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 127 k 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:18.736 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:18.737 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:18.737 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 127 k 2024-03-30T07:04:18.737 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 456 k 2024-03-30T07:04:18.737 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:18.965 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-18.2.2-1055.g55c4b44e.el9.x86_64 558 kB/s | 127 kB 00:00 2024-03-30T07:04:18.965 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:18.965 INFO:teuthology.orchestra.run.smithi047.stdout:Total 554 kB/s | 127 kB 00:00 2024-03-30T07:04:18.965 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:18.971 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:18.971 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:19.028 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:19.028 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:19.175 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:19.707 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:20.062 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:20.337 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2 2024-03-30T07:04:20.913 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:21.015 INFO:teuthology.orchestra.run.smithi047.stdout:Package libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:21.070 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:21.071 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:21.072 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:21.134 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs-devel 2024-03-30T07:04:21.702 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:33 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:21.858 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 31 k 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:21.859 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 31 k 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 139 k 2024-03-30T07:04:21.860 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:22.093 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs-devel-18.2.2-1055.g55c4b44e.el9.x86_6 133 kB/s | 31 kB 00:00 2024-03-30T07:04:22.094 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:22.094 INFO:teuthology.orchestra.run.smithi047.stdout:Total 132 kB/s | 31 kB 00:00 2024-03-30T07:04:22.094 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:22.097 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:22.098 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:22.115 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:22.115 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:22.206 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:22.327 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:22.730 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:22.987 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:23.186 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2 2024-03-30T07:04:23.758 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:23.859 INFO:teuthology.orchestra.run.smithi047.stdout:Package librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:23.914 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:23.916 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:23.916 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:23.984 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1 2024-03-30T07:04:24.556 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:24.657 INFO:teuthology.orchestra.run.smithi047.stdout:Package librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:24.712 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:24.713 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:24.713 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:24.775 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados 2024-03-30T07:04:25.348 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:25.449 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:25.504 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:25.505 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:25.505 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:25.570 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw 2024-03-30T07:04:26.138 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:26.240 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:26.295 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:26.296 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:26.296 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:26.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs 2024-03-30T07:04:26.934 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:27.037 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:27.094 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:27.095 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:27.096 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:27.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd 2024-03-30T07:04:27.753 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:27.855 INFO:teuthology.orchestra.run.smithi047.stdout:Package python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:04:27.911 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:27.912 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:04:27.912 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:27.974 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse 2024-03-30T07:04:28.546 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:28.702 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 87 k 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:28.703 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:28.704 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:28.704 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 87 k 2024-03-30T07:04:28.704 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 227 k 2024-03-30T07:04:28.704 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:29.088 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 225 kB/s | 87 kB 00:00 2024-03-30T07:04:29.089 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:29.089 INFO:teuthology.orchestra.run.smithi047.stdout:Total 225 kB/s | 87 kB 00:00 2024-03-30T07:04:29.089 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:29.097 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:29.097 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:29.155 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:29.155 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:29.302 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:29.405 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:29.758 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:30.038 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:30.271 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror 2024-03-30T07:04:30.843 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:31.000 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.0 M 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 3.0 M 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 12 M 2024-03-30T07:04:31.001 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:31.575 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-30T07:04:31.575 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:31.576 INFO:teuthology.orchestra.run.smithi047.stdout:Total 5.2 MB/s | 3.0 MB 00:00 2024-03-30T07:04:31.576 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:31.585 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:31.586 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:31.660 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:31.661 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:32.090 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:32.150 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T07:04:32.175 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:33.327 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:33.328 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:33.328 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:33.328 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:33.328 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:33.328 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:33.576 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd 2024-03-30T07:04:34.152 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:34.309 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:34.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:34.310 INFO:teuthology.orchestra.run.smithi047.stdout: Package Architecture Version Repository Size 2024-03-30T07:04:34.310 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:34.310 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 172 k 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 172 k 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 494 k 2024-03-30T07:04:34.311 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:34.662 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 490 kB/s | 172 kB 00:00 2024-03-30T07:04:34.663 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:34.663 INFO:teuthology.orchestra.run.smithi047.stdout:Total 488 kB/s | 172 kB 00:00 2024-03-30T07:04:34.663 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:34.672 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:34.673 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:34.730 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:34.731 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:34.878 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:34.988 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:35.462 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:35.736 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:04:35.736 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:35.737 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:35.737 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:35.737 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:35.737 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:36.002 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base-debuginfo 2024-03-30T07:04:36.573 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:36.730 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 67 M 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout:Installing dependencies: 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 20 M 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout:Install 2 Packages 2024-03-30T07:04:36.731 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:36.732 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 87 M 2024-03-30T07:04:36.732 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 482 M 2024-03-30T07:04:36.732 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:04:38.057 INFO:teuthology.orchestra.run.smithi047.stdout:(1/2): ceph-debuginfo-18.2.2-1055.g55c4b44e.el9 15 MB/s | 20 MB 00:01 2024-03-30T07:04:42.124 INFO:teuthology.orchestra.run.smithi047.stdout:(2/2): ceph-base-debuginfo-18.2.2-1055.g55c4b44 12 MB/s | 67 MB 00:05 2024-03-30T07:04:42.125 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:04:42.126 INFO:teuthology.orchestra.run.smithi047.stdout:Total 16 MB/s | 87 MB 00:05 2024-03-30T07:04:42.126 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:04:42.131 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:04:42.131 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:04:42.724 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:04:42.724 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:04:45.752 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:04:53.152 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/2 2024-03-30T07:04:53.227 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 2/2 2024-03-30T07:04:56.354 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 2/2 2024-03-30T07:04:56.354 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/2 2024-03-30T07:04:56.616 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:56.617 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:04:56.860 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-common-debuginfo 2024-03-30T07:04:57.431 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:02:09 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:04:57.587 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:04:57.588 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:57.588 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:04:57.588 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:57.588 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:04:57.588 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 399 M 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 399 M 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.6 G 2024-03-30T07:04:57.589 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:05:16.420 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-common-debuginfo-18.2.2-1055.g55c4b44e.el9 21 MB/s | 399 MB 00:18 2024-03-30T07:05:16.421 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:05:16.421 INFO:teuthology.orchestra.run.smithi047.stdout:Total 21 MB/s | 399 MB 00:18 2024-03-30T07:05:16.421 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:05:16.425 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:05:16.425 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:05:18.911 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:05:18.912 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:06:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:06:01.718 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:06:09.032 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:06:09.289 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:06:09.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:09.290 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:06:09.290 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:06:09.290 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:09.290 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:06:09.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-30T07:06:10.101 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:03:22 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:06:10.261 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:06:10.261 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:06:10.261 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.4 M 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:======================================================================================== 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.4 M 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 6.2 M 2024-03-30T07:06:10.262 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:06:10.753 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-10 2.9 MB/s | 1.4 MB 00:00 2024-03-30T07:06:10.754 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:06:10.754 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.9 MB/s | 1.4 MB 00:00 2024-03-30T07:06:10.754 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:06:10.758 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:06:10.758 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:06:10.801 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:06:10.801 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:06:11.065 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:06:11.165 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:06:11.570 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:06:11.882 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:06:11.882 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:11.882 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:06:11.882 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:06:11.883 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:11.883 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:06:12.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-30T07:06:12.619 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:03:24 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:06:12.776 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 162 M 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:12.777 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 162 M 2024-03-30T07:06:12.778 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 679 M 2024-03-30T07:06:12.778 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:06:18.069 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 86 kB/s | 19 MB 03:44 2024-03-30T07:06:19.922 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-radosgw-debuginfo-18.2.2-1055.g55c4b44e.el 23 MB/s | 162 MB 00:07 2024-03-30T07:06:19.923 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:06:19.923 INFO:teuthology.orchestra.run.smithi047.stdout:Total 23 MB/s | 162 MB 00:07 2024-03-30T07:06:19.924 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:06:19.928 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:06:19.928 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:06:20.958 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:06:20.959 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:06:23.311 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-30T07:06:24.762 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-03-30T07:06:27.664 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-30T07:06:32.747 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-30T07:06:35.422 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 7.7 M 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.3 M 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.0 M 2024-03-30T07:06:35.436 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.1 M 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 18 M 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 25 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 709 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T07:06:35.437 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 474 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 4.5 M 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 45 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 129 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 162 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 320 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 298 k 2024-03-30T07:06:35.438 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 99 k 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout:Install 28 Packages 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-03-30T07:06:35.439 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:06:35.440 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 53 M 2024-03-30T07:06:35.440 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:06:37.515 INFO:teuthology.orchestra.run.smithi052.stdout:(1/30): ceph-common-18.2.2-1055.g55c4b44e.el9.x 11 MB/s | 18 MB 00:01 2024-03-30T07:06:37.565 INFO:teuthology.orchestra.run.smithi052.stdout:(2/30): ceph-selinux-18.2.2-1055.g55c4b44e.el9. 506 kB/s | 25 kB 00:00 2024-03-30T07:06:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:(3/30): libcephfs2-18.2.2-1055.g55c4b44e.el9.x8 10 MB/s | 709 kB 00:00 2024-03-30T07:06:37.690 INFO:teuthology.orchestra.run.smithi052.stdout:(4/30): libradosstriper1-18.2.2-1055.g55c4b44e. 8.0 MB/s | 474 kB 00:00 2024-03-30T07:06:37.966 INFO:teuthology.orchestra.run.smithi052.stdout:(5/30): librgw2-18.2.2-1055.g55c4b44e.el9.x86_6 16 MB/s | 4.5 MB 00:00 2024-03-30T07:06:38.008 INFO:teuthology.orchestra.run.smithi052.stdout:(6/30): python3-ceph-argparse-18.2.2-1055.g55c4 1.1 MB/s | 45 kB 00:00 2024-03-30T07:06:38.058 INFO:teuthology.orchestra.run.smithi052.stdout:(7/30): python3-ceph-common-18.2.2-1055.g55c4b4 2.5 MB/s | 129 kB 00:00 2024-03-30T07:06:38.108 INFO:teuthology.orchestra.run.smithi052.stdout:(8/30): python3-cephfs-18.2.2-1055.g55c4b44e.el 3.2 MB/s | 162 kB 00:00 2024-03-30T07:06:38.167 INFO:teuthology.orchestra.run.smithi052.stdout:(9/30): python3-rados-18.2.2-1055.g55c4b44e.el9 5.4 MB/s | 320 kB 00:00 2024-03-30T07:06:38.234 INFO:teuthology.orchestra.run.smithi052.stdout:(10/30): ceph-base-18.2.2-1055.g55c4b44e.el9.x8 2.1 MB/s | 5.1 MB 00:02 2024-03-30T07:06:38.260 INFO:teuthology.orchestra.run.smithi052.stdout:(11/30): python3-rbd-18.2.2-1055.g55c4b44e.el9. 3.2 MB/s | 298 kB 00:00 2024-03-30T07:06:38.326 INFO:teuthology.orchestra.run.smithi052.stdout:(12/30): python3-rgw-18.2.2-1055.g55c4b44e.el9. 1.1 MB/s | 99 kB 00:00 2024-03-30T07:06:38.435 INFO:teuthology.orchestra.run.smithi052.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-30T07:06:38.469 INFO:teuthology.orchestra.run.smithi052.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-30T07:06:38.511 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:06:38.536 INFO:teuthology.orchestra.run.smithi052.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-30T07:06:38.594 INFO:teuthology.orchestra.run.smithi052.stdout:(16/30): boost-program-options-1.75.0-8.el9.x86 316 kB/s | 106 kB 00:00 2024-03-30T07:06:38.613 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:06:38.628 INFO:teuthology.orchestra.run.smithi052.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 282 kB 00:00 2024-03-30T07:06:38.653 INFO:teuthology.orchestra.run.smithi052.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 712 kB/s | 41 kB 00:00 2024-03-30T07:06:39.278 INFO:teuthology.orchestra.run.smithi052.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 170 kB/s | 111 kB 00:00 2024-03-30T07:06:39.570 INFO:teuthology.orchestra.run.smithi052.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 337 kB/s | 309 kB 00:00 2024-03-30T07:06:39.779 INFO:teuthology.orchestra.run.smithi052.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-03-30T07:06:39.937 INFO:teuthology.orchestra.run.smithi052.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 307 kB/s | 49 kB 00:00 2024-03-30T07:06:40.179 INFO:teuthology.orchestra.run.smithi052.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 279 kB/s | 67 kB 00:00 2024-03-30T07:06:40.396 INFO:teuthology.orchestra.run.smithi052.stdout:(24/30): ceph-radosgw-18.2.2-1055.g55c4b44e.el9 1.7 MB/s | 7.7 MB 00:04 2024-03-30T07:06:41.030 INFO:teuthology.orchestra.run.smithi052.stdout:(25/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 986 kB/s | 838 kB 00:00 2024-03-30T07:06:41.172 INFO:teuthology.orchestra.run.smithi052.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2024-03-30T07:06:41.272 INFO:teuthology.orchestra.run.smithi052.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 219 kB/s | 192 kB 00:00 2024-03-30T07:06:41.539 INFO:teuthology.orchestra.run.smithi052.stdout:(28/30): librbd1-18.2.2-1055.g55c4b44e.el9.x86_ 11 MB/s | 3.0 MB 00:00 2024-03-30T07:06:41.806 INFO:teuthology.orchestra.run.smithi052.stdout:(29/30): librados2-18.2.2-1055.g55c4b44e.el9.x8 5.2 MB/s | 3.3 MB 00:00 2024-03-30T07:06:42.431 INFO:teuthology.orchestra.run.smithi052.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-03-30T07:06:42.439 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:06:42.439 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.6 MB/s | 53 MB 00:07 2024-03-30T07:06:42.439 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:06:42.480 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:06:42.480 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:06:42.741 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:06:42.937 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:06:42.938 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:42.996 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:06:43.200 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-test-debuginfo 2024-03-30T07:06:43.773 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:03:55 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:06:43.930 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:06:43.930 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:43.930 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:06:43.930 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 582 M 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 582 M 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.6 G 2024-03-30T07:06:43.931 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:06:44.410 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:06:45.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-30T07:06:45.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-30T07:06:45.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-30T07:06:45.417 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9. 9/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 10/32 2024-03-30T07:06:45.418 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 11/32 2024-03-30T07:06:45.420 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 11/32 2024-03-30T07:06:45.531 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-30T07:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-30T07:06:45.603 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-30T07:06:45.636 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-30T07:06:45.663 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-30T07:06:45.708 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-30T07:06:45.739 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-30T07:06:46.054 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-30T07:06:46.078 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 20/32 2024-03-30T07:06:46.152 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 20/32 2024-03-30T07:06:46.189 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 21/32 2024-03-30T07:06:46.237 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-30T07:06:46.662 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-30T07:06:46.697 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 24/32 2024-03-30T07:06:46.768 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 24/32 2024-03-30T07:06:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 25/32 2024-03-30T07:06:46.866 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x8 26/32 2024-03-30T07:06:49.011 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:06:49.046 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:06:49.595 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 27/32 2024-03-30T07:06:49.641 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 28/32 2024-03-30T07:06:49.713 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 28/32 2024-03-30T07:06:49.713 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T07:06:49.713 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T07:06:49.714 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:06:49.759 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-03-30T07:06:59.627 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-03-30T07:06:59.628 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-03-30T07:06:59.628 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:00.409 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/32 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/32 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T07:07:00.433 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:00.435 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-30T07:07:00.521 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-30T07:07:00.521 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:07:01.607 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 3/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 4/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_6 6/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 7/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9. 8/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x8 9/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 10/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 11/32 2024-03-30T07:07:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 12/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 13/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-30T07:07:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-30T07:07:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-30T07:07:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-30T07:07:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/32 2024-03-30T07:07:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-30T07:07:01.611 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/32 2024-03-30T07:07:02.691 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-30T07:07:02.691 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:02.691 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T07:07:02.692 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T07:07:02.693 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:02.694 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:03.048 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-03-30T07:07:03.621 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:31 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:03.786 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:03.787 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:03.787 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:03.787 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:03.787 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 40 M 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 166 k 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-30T07:07:03.788 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:03.789 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 41 M 2024-03-30T07:07:03.789 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 164 M 2024-03-30T07:07:03.789 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:04.050 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-30T07:07:04.117 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-30T07:07:04.151 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-30T07:07:04.318 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-30T07:07:04.626 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-18.2.2-1055.g55c4b44e.el9. 216 kB/s | 166 kB 00:00 2024-03-30T07:07:10.026 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-18.2.2-1055.g55c4b44e.el9.x86_ 6.4 MB/s | 40 MB 00:06 2024-03-30T07:07:10.029 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:10.029 INFO:teuthology.orchestra.run.smithi052.stdout:Total 6.5 MB/s | 41 MB 00:06 2024-03-30T07:07:10.043 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:10.063 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:10.063 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:10.416 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:10.416 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:10.964 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:11.062 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T07:07:11.111 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T07:07:11.150 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T07:07:11.196 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T07:07:11.223 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/6 2024-03-30T07:07:15.134 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/6 2024-03-30T07:07:15.334 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 6/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T07:07:17.310 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T07:07:17.404 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-test-debuginfo-18.2.2-1055.g55c4b44e.el9.x 17 MB/s | 582 MB 00:33 2024-03-30T07:07:17.405 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:17.406 INFO:teuthology.orchestra.run.smithi047.stdout:Total 17 MB/s | 582 MB 00:33 2024-03-30T07:07:17.406 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:07:17.411 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:07:17.414 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:07:17.623 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T07:07:17.623 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:17.623 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:17.623 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:17.623 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.6-16.el9.x86_64 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:17.624 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:17.882 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-03-30T07:07:18.460 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:46 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:18.629 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 6.5 k 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 2.1 M 2024-03-30T07:07:18.632 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.5 M 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 246 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 4.7 M 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 17 M 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T07:07:18.633 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T07:07:18.634 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout:Install 33 Packages 2024-03-30T07:07:18.635 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:18.636 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 29 M 2024-03-30T07:07:18.636 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 104 M 2024-03-30T07:07:18.636 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:19.480 INFO:teuthology.orchestra.run.smithi052.stdout:(1/33): ceph-18.2.2-1055.g55c4b44e.el9.x86_64.r 29 kB/s | 6.5 kB 00:00 2024-03-30T07:07:19.613 INFO:teuthology.orchestra.run.smithi052.stdout:(2/33): ceph-mgr-18.2.2-1055.g55c4b44e.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-03-30T07:07:19.980 INFO:teuthology.orchestra.run.smithi052.stdout:(3/33): ceph-mds-18.2.2-1055.g55c4b44e.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-30T07:07:20.138 INFO:teuthology.orchestra.run.smithi052.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1055.g55c4 1.5 MB/s | 246 kB 00:00 2024-03-30T07:07:20.256 INFO:teuthology.orchestra.run.smithi052.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-30T07:07:20.289 INFO:teuthology.orchestra.run.smithi052.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-30T07:07:20.323 INFO:teuthology.orchestra.run.smithi052.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-30T07:07:20.357 INFO:teuthology.orchestra.run.smithi052.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-30T07:07:20.432 INFO:teuthology.orchestra.run.smithi052.stdout:(9/33): ceph-mon-18.2.2-1055.g55c4b44e.el9.x86_ 4.9 MB/s | 4.7 MB 00:00 2024-03-30T07:07:20.457 INFO:teuthology.orchestra.run.smithi052.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2024-03-30T07:07:20.529 INFO:teuthology.orchestra.run.smithi052.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-03-30T07:07:20.549 INFO:teuthology.orchestra.run.smithi052.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-30T07:07:20.574 INFO:teuthology.orchestra.run.smithi052.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-30T07:07:20.600 INFO:teuthology.orchestra.run.smithi052.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-30T07:07:20.776 INFO:teuthology.orchestra.run.smithi052.stdout:(15/33): ceph-osd-18.2.2-1055.g55c4b44e.el9.x86 14 MB/s | 17 MB 00:01 2024-03-30T07:07:20.951 INFO:teuthology.orchestra.run.smithi052.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 332 kB/s | 125 kB 00:00 2024-03-30T07:07:21.012 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:07:21.012 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:07:21.084 INFO:teuthology.orchestra.run.smithi052.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 90 kB/s | 43 kB 00:00 2024-03-30T07:07:21.187 INFO:teuthology.orchestra.run.smithi052.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-30T07:07:21.234 INFO:teuthology.orchestra.run.smithi052.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 374 kB/s | 172 kB 00:00 2024-03-30T07:07:21.294 INFO:teuthology.orchestra.run.smithi052.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-30T07:07:21.335 INFO:teuthology.orchestra.run.smithi052.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-30T07:07:21.402 INFO:teuthology.orchestra.run.smithi052.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-30T07:07:21.435 INFO:teuthology.orchestra.run.smithi052.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-30T07:07:21.485 INFO:teuthology.orchestra.run.smithi052.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 670 kB/s | 358 kB 00:00 2024-03-30T07:07:21.535 INFO:teuthology.orchestra.run.smithi052.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 346 kB/s | 46 kB 00:00 2024-03-30T07:07:21.562 INFO:teuthology.orchestra.run.smithi052.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-03-30T07:07:21.619 INFO:teuthology.orchestra.run.smithi052.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-30T07:07:21.645 INFO:teuthology.orchestra.run.smithi052.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-30T07:07:21.678 INFO:teuthology.orchestra.run.smithi052.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 768 kB/s | 90 kB 00:00 2024-03-30T07:07:21.712 INFO:teuthology.orchestra.run.smithi052.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-03-30T07:07:21.804 INFO:teuthology.orchestra.run.smithi052.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-30T07:07:21.906 INFO:teuthology.orchestra.run.smithi052.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 887 kB/s | 230 kB 00:00 2024-03-30T07:07:21.929 INFO:teuthology.orchestra.run.smithi052.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-30T07:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:21.933 INFO:teuthology.orchestra.run.smithi052.stdout:Total 8.8 MB/s | 29 MB 00:03 2024-03-30T07:07:21.947 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:21.991 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:22.349 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:22.350 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:23.669 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:23.741 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-30T07:07:23.773 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-30T07:07:23.814 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-30T07:07:23.912 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-30T07:07:23.987 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-30T07:07:24.034 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-30T07:07:24.108 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-30T07:07:24.160 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-30T07:07:24.206 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-30T07:07:24.256 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-30T07:07:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-30T07:07:24.526 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-30T07:07:24.576 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-30T07:07:24.655 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-30T07:07:24.691 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-30T07:07:24.721 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-30T07:07:24.801 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-30T07:07:24.831 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-30T07:07:24.864 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-30T07:07:24.905 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-30T07:07:24.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-30T07:07:24.974 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-30T07:07:25.068 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-30T07:07:25.105 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-30T07:07:25.180 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-30T07:07:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-30T07:07:25.451 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-30T07:07:25.605 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9. 28/33 2024-03-30T07:07:25.642 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/33 2024-03-30T07:07:25.667 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 29/33 2024-03-30T07:07:25.668 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:25.668 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T07:07:25.668 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T07:07:25.668 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T07:07:25.668 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:27.160 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/33 2024-03-30T07:07:27.186 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 30/33 2024-03-30T07:07:27.186 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:27.186 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T07:07:27.186 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T07:07:27.186 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T07:07:27.187 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:27.652 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/33 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 31/33 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T07:07:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:27.886 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 32/33 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 32/33 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T07:07:27.909 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:27.986 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 33/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 33/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 3/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 4/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 5/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9. 6/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-30T07:07:29.683 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-30T07:07:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-30T07:07:29.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-30T07:07:30.695 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-30T07:07:30.695 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T07:07:30.696 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T07:07:30.697 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:30.698 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:31.029 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-03-30T07:07:31.601 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:31.703 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:07:31.757 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:31.758 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:07:31.758 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:31.818 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-03-30T07:07:32.382 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:00 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:32.536 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:32.537 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:32.537 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:32.538 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:32.538 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:32.538 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 224 k 2024-03-30T07:07:32.538 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 224 k 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 216 k 2024-03-30T07:07:32.541 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:32.890 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-18.2.2-1055.g55c4b44e.el9.noarch.rpm 638 kB/s | 224 kB 00:00 2024-03-30T07:07:32.891 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:32.891 INFO:teuthology.orchestra.run.smithi052.stdout:Total 635 kB/s | 224 kB 00:00 2024-03-30T07:07:32.891 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:32.896 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:32.896 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:32.899 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:32.899 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:32.975 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:33.376 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:07:33.455 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:07:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:07:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/1 2024-03-30T07:07:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:34.098 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:34.098 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:34.099 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:34.099 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:34.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T07:07:34.891 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:35.050 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 143 k 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 143 k 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 395 k 2024-03-30T07:07:35.051 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:35.278 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-18.2.2-1055.g55c4b4 630 kB/s | 143 kB 00:00 2024-03-30T07:07:35.279 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:35.279 INFO:teuthology.orchestra.run.smithi052.stdout:Total 626 kB/s | 143 kB 00:00 2024-03-30T07:07:35.279 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:35.287 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:35.287 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:35.341 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:35.341 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:35.621 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:35.687 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:07:35.711 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:07:35.712 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:07:35.712 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T07:07:35.712 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e. 1/1 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:36.687 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:36.961 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-03-30T07:07:37.533 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:05 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:37.634 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:07:37.689 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:37.690 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:07:37.690 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:37.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T07:07:38.328 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:06 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:38.484 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:38.484 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 3.5 M 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 23 k 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 15 k 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:38.485 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:07:38.486 INFO:teuthology.orchestra.run.smithi052.stdout:Install 5 Packages 2024-03-30T07:07:38.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:38.486 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.8 M 2024-03-30T07:07:38.486 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 87 M 2024-03-30T07:07:38.486 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:38.748 INFO:teuthology.orchestra.run.smithi052.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1055.g55c4 78 kB/s | 15 kB 00:00 2024-03-30T07:07:38.798 INFO:teuthology.orchestra.run.smithi052.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1055.g55c 97 kB/s | 23 kB 00:00 2024-03-30T07:07:39.174 INFO:teuthology.orchestra.run.smithi052.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 72 kB/s | 31 kB 00:00 2024-03-30T07:07:39.382 INFO:teuthology.orchestra.run.smithi052.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1055.g55c4b44e 4.3 MB/s | 3.5 MB 00:00 2024-03-30T07:07:39.610 INFO:teuthology.orchestra.run.smithi052.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 232 kB/s | 188 kB 00:00 2024-03-30T07:07:39.613 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:39.613 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.4 MB/s | 3.8 MB 00:01 2024-03-30T07:07:39.614 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:39.621 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:39.621 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:39.700 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:39.700 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:39.958 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:40.065 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-30T07:07:40.095 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-30T07:07:40.124 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.n 3/5 2024-03-30T07:07:41.239 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9. 4/5 2024-03-30T07:07:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 5/5 2024-03-30T07:07:42.364 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 5/5 2024-03-30T07:07:42.365 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9. 1/5 2024-03-30T07:07:42.365 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarc 2/5 2024-03-30T07:07:42.365 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.n 3/5 2024-03-30T07:07:42.365 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:42.671 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:42.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T07:07:43.459 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:11 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:43.617 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:43.618 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-30T07:07:43.618 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 7.4 M 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T07:07:43.619 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:43.620 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2024-03-30T07:07:43.621 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2024-03-30T07:07:43.621 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:44.005 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 202 kB/s | 32 kB 00:00 2024-03-30T07:07:44.039 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-30T07:07:44.073 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-30T07:07:44.223 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2024-03-30T07:07:44.357 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-30T07:07:44.424 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-30T07:07:44.449 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-30T07:07:44.658 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): ceph-mgr-diskprediction-local-18.2.2-10 9.1 MB/s | 7.4 MB 00:00 2024-03-30T07:07:44.825 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-30T07:07:44.900 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-30T07:07:44.983 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 605 kB/s | 195 kB 00:00 2024-03-30T07:07:45.075 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-03-30T07:07:45.078 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:45.078 INFO:teuthology.orchestra.run.smithi052.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-30T07:07:45.118 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:45.142 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:45.142 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:45.618 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:45.618 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:46.334 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T07:07:46.429 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T07:07:46.527 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T07:07:46.574 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T07:07:46.769 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T07:07:46.806 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T07:07:46.890 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T07:07:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T07:07:47.697 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T07:07:49.022 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T07:07:50.092 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T07:07:50.121 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 12/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 12/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b 1/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T07:07:52.699 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T07:07:52.700 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T07:07:53.164 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:53.165 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:53.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-03-30T07:07:54.126 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:22 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:54.286 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 49 k 2024-03-30T07:07:54.287 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T07:07:54.288 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-03-30T07:07:54.289 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:07:54.505 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-03-30T07:07:54.530 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-30T07:07:54.589 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-30T07:07:54.614 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-30T07:07:54.639 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-30T07:07:54.664 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-30T07:07:54.690 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): ceph-mgr-rook-18.2.2-1055.g55c4b44e.el9 153 kB/s | 49 kB 00:00 2024-03-30T07:07:54.932 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-30T07:07:54.957 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-30T07:07:55.107 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-30T07:07:55.133 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 458 kB/s | 203 kB 00:00 2024-03-30T07:07:55.233 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-03-30T07:07:55.266 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-30T07:07:55.271 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:07:55.271 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-30T07:07:55.271 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:07:55.285 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:07:55.286 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:07:55.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:07:55.383 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:07:55.902 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:07:56.072 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T07:07:56.127 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T07:07:56.184 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T07:07:56.218 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T07:07:56.261 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T07:07:56.359 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T07:07:56.468 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T07:07:56.521 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T07:07:57.295 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T07:07:57.341 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T07:07:57.375 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T07:07:57.423 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T07:07:57.453 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 13/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 13/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 1/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T07:07:58.293 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T07:07:58.294 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T07:07:58.653 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T07:07:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T07:07:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T07:07:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T07:07:58.654 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:58.654 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:07:58.912 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T07:07:59.482 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:27 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:07:59.639 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:07:59.640 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:59.640 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:07:59.640 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:59.640 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:07:59.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 138 k 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-30T07:07:59.641 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:07:59.642 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.8 M 2024-03-30T07:07:59.642 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-03-30T07:07:59.642 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:00.292 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-30T07:08:00.334 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1055.g55c4b44e.e 452 kB/s | 138 kB 00:00 2024-03-30T07:08:00.426 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 630 kB/s | 58 kB 00:00 2024-03-30T07:08:00.468 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-30T07:08:00.518 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.7 MB/s | 548 kB 00:00 2024-03-30T07:08:00.968 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 264 kB/s | 248 kB 00:00 2024-03-30T07:08:00.970 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:00.970 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-03-30T07:08:00.985 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:00.996 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:00.997 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:01.093 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:01.095 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:01.403 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:01.574 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T07:08:01.638 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T07:08:02.204 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T07:08:02.293 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T07:08:02.361 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T07:08:02.390 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 6/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 6/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 1/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T07:08:03.179 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T07:08:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T07:08:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T07:08:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:03.485 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:03.749 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-03-30T07:08:04.319 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:04.476 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 849 k 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-30T07:08:04.477 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:04.478 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 927 k 2024-03-30T07:08:04.478 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2024-03-30T07:08:04.478 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:04.611 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 590 kB/s | 79 kB 00:00 2024-03-30T07:08:04.936 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-18.2.2-1055.g55c4b44e.el9.x86_ 1.8 MB/s | 849 kB 00:00 2024-03-30T07:08:04.938 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:04.938 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.0 MB/s | 927 kB 00:00 2024-03-30T07:08:04.938 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:04.959 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:04.959 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:05.034 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:05.034 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:05.232 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:05.372 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T07:08:05.439 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/2 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:06.492 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:06.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-03-30T07:08:07.355 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:35 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:07.512 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:07.513 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:07.513 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-30T07:08:07.513 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:07.513 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:07.513 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:18.2.2-1055.g55c4b44e.el9 ceph-noarch 264 k 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 332 k 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.5 M 2024-03-30T07:08:07.514 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:07.640 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 546 kB/s | 69 kB 00:00 2024-03-30T07:08:07.915 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-volume-18.2.2-1055.g55c4b44e.el9.no 658 kB/s | 264 kB 00:00 2024-03-30T07:08:07.916 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:07.917 INFO:teuthology.orchestra.run.smithi052.stdout:Total 824 kB/s | 332 kB 00:00 2024-03-30T07:08:07.917 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:07.927 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:07.927 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:07.969 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:07.969 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:08.116 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:08.299 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T07:08:08.326 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2/2 2024-03-30T07:08:08.342 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2/2 2024-03-30T07:08:08.342 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:08:08.342 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T07:08:08.342 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:09.065 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 1/2 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:18.2.2-1055.g55c4b44e.el9.noarch 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:09.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-03-30T07:08:10.285 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:10.442 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:10.443 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:10.443 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:08:10.443 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 127 k 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 127 k 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-03-30T07:08:10.444 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:10.653 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-18.2.2-1055.g55c4b44e.el9.x86_64 606 kB/s | 127 kB 00:00 2024-03-30T07:08:10.654 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:10.654 INFO:teuthology.orchestra.run.smithi052.stdout:Total 602 kB/s | 127 kB 00:00 2024-03-30T07:08:10.654 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:10.660 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:10.660 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:10.718 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:10.719 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:10.862 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:10.962 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:11.429 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:11.716 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:11.717 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:11.717 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:11.717 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:11.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-03-30T07:08:12.530 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:12.631 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:12.687 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:12.688 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:12.688 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:12.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-03-30T07:08:13.327 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:13.484 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:13.485 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:13.485 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:08:13.485 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:13.485 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 31 k 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 k 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 139 k 2024-03-30T07:08:13.486 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:13.675 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-18.2.2-1055.g55c4b44e.el9.x86_6 164 kB/s | 31 kB 00:00 2024-03-30T07:08:13.676 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:13.676 INFO:teuthology.orchestra.run.smithi052.stdout:Total 163 kB/s | 31 kB 00:00 2024-03-30T07:08:13.676 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:13.680 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:13.680 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:13.701 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:13.702 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:13.785 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:13.919 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:14.308 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:14.565 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:14.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:14.566 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:14.566 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:14.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:14.566 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:14.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-03-30T07:08:15.322 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:15.423 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:15.478 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:15.479 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:15.479 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:15.545 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-03-30T07:08:16.116 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:16.217 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:16.272 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:16.273 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:16.273 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:16.350 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-03-30T07:08:16.920 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:17.023 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:17.078 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:17.096 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:17.097 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:17.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-03-30T07:08:17.723 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:17.824 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:17.879 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:17.880 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:17.880 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:17.949 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-03-30T07:08:18.521 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:18.623 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:18.678 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:18.679 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:18.679 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:18.750 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-03-30T07:08:19.323 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:47 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:19.426 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:19.481 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:19.482 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:08:19.482 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:19.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-03-30T07:08:20.127 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:20.285 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 87 k 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:20.286 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:20.287 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 87 k 2024-03-30T07:08:20.287 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 227 k 2024-03-30T07:08:20.287 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:20.712 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 204 kB/s | 87 kB 00:00 2024-03-30T07:08:20.712 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:20.712 INFO:teuthology.orchestra.run.smithi052.stdout:Total 203 kB/s | 87 kB 00:00 2024-03-30T07:08:20.713 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:20.720 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:20.721 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:20.778 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:20.778 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:20.921 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:21.781 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:21.782 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:21.782 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:21.782 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:22.036 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-03-30T07:08:22.618 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:22.778 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:22.779 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:22.779 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:08:22.779 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:22.779 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 3.0 M 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.0 M 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 12 M 2024-03-30T07:08:22.780 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:23.173 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2024-03-30T07:08:23.174 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:23.174 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.6 MB/s | 3.0 MB 00:00 2024-03-30T07:08:23.174 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:23.184 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:23.184 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:23.260 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:23.260 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:23.703 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:23.766 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T07:08:23.791 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:24.967 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:25.194 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-03-30T07:08:25.501 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:08:25.608 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:08:25.773 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:25.930 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 172 k 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:25.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:25.932 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:25.932 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:25.932 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 172 k 2024-03-30T07:08:25.932 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 494 k 2024-03-30T07:08:25.932 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:26.434 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-18.2.2-1055.g55c4b44e.el9.x86_64.rpm 342 kB/s | 172 kB 00:00 2024-03-30T07:08:26.435 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:26.435 INFO:teuthology.orchestra.run.smithi052.stdout:Total 341 kB/s | 172 kB 00:00 2024-03-30T07:08:26.435 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:26.444 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:26.445 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:26.503 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:26.503 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:26.657 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:26.760 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:27.188 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:27.482 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:27.735 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2024-03-30T07:08:28.307 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:56 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:28.463 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 67 M 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 20 M 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:28.464 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-30T07:08:28.465 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:28.465 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 87 M 2024-03-30T07:08:28.465 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 482 M 2024-03-30T07:08:28.465 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:31.346 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): ceph-debuginfo-18.2.2-1055.g55c4b44e.el9 7.1 MB/s | 20 MB 00:02 2024-03-30T07:08:33.404 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-base-debuginfo-18.2.2-1055.g55c4b44 14 MB/s | 67 MB 00:04 2024-03-30T07:08:33.406 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:33.406 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 87 MB 00:04 2024-03-30T07:08:33.406 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:08:33.411 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:08:33.411 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:08:33.631 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:33.889 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:08:34.007 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:08:34.007 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:08:34.160 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-base-debuginfo 2024-03-30T07:08:34.730 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:46 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:08:34.829 INFO:teuthology.orchestra.run.smithi047.stdout:Package ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:08:34.883 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:08:34.884 INFO:teuthology.orchestra.run.smithi047.stdout:Nothing to do. 2024-03-30T07:08:34.885 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:08:34.955 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mgr-debuginfo 2024-03-30T07:08:35.524 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:47 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:08:35.678 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 19 M 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:35.679 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:08:35.680 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:35.680 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 19 M 2024-03-30T07:08:35.680 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 101 M 2024-03-30T07:08:35.680 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:08:37.022 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:08:37.117 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mgr-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 13 MB/s | 19 MB 00:01 2024-03-30T07:08:37.117 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:37.118 INFO:teuthology.orchestra.run.smithi047.stdout:Total 13 MB/s | 19 MB 00:01 2024-03-30T07:08:37.118 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:08:37.122 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:08:37.122 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:08:37.274 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:08:37.274 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:08:39.587 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:08:39.701 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:40.661 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:40.924 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:40.924 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:40.924 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:08:40.924 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:40.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:40.925 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:08:41.089 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mds-debuginfo 2024-03-30T07:08:41.661 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:05:53 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:08:41.816 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 47 M 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:41.817 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:08:41.818 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:41.818 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 47 M 2024-03-30T07:08:41.818 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 171 M 2024-03-30T07:08:41.818 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:08:44.095 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mds-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 21 MB/s | 47 MB 00:02 2024-03-30T07:08:44.096 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:08:44.096 INFO:teuthology.orchestra.run.smithi047.stdout:Total 21 MB/s | 47 MB 00:02 2024-03-30T07:08:44.096 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:08:44.100 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:08:44.100 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:08:44.425 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:08:44.425 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:08:44.437 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/2 2024-03-30T07:08:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 2/2 2024-03-30T07:08:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 2/2 2024-03-30T07:08:47.671 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/2 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2/2 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:47.961 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:47.962 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:08:48.200 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-common-debuginfo 2024-03-30T07:08:48.775 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:16 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:08:48.932 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:08:48.932 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:48.932 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 399 M 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 399 M 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.6 G 2024-03-30T07:08:48.933 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:08:49.402 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:08:49.503 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:50.831 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:51.098 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:08:51.279 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-mon-debuginfo 2024-03-30T07:08:51.852 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:06:03 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:08:52.007 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 98 M 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:52.008 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 98 M 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 450 M 2024-03-30T07:08:52.009 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:09:04.154 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-mon-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 8.1 MB/s | 98 MB 00:12 2024-03-30T07:09:04.155 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:09:04.155 INFO:teuthology.orchestra.run.smithi047.stdout:Total 8.1 MB/s | 98 MB 00:12 2024-03-30T07:09:04.156 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:09:04.160 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:09:04.160 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:09:04.798 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:09:04.798 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:09:14.674 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-common-debuginfo-18.2.2-1055.g55c4b44e.el9 16 MB/s | 399 MB 00:25 2024-03-30T07:09:14.674 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:09:14.675 INFO:teuthology.orchestra.run.smithi052.stdout:Total 16 MB/s | 399 MB 00:25 2024-03-30T07:09:14.675 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:09:14.679 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:09:14.679 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:09:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:09:15.763 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:09:17.168 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:09:17.168 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:09:18.671 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:09:18.934 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:09:19.113 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-osd-debuginfo 2024-03-30T07:09:19.684 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:06:31 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:09:19.840 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 286 M 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:09:19.841 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 286 M 2024-03-30T07:09:19.842 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.2 G 2024-03-30T07:09:19.842 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:09:32.547 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-osd-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 23 MB/s | 286 MB 00:12 2024-03-30T07:09:32.548 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:09:32.548 INFO:teuthology.orchestra.run.smithi047.stdout:Total 23 MB/s | 286 MB 00:12 2024-03-30T07:09:32.549 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:09:32.553 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:09:32.553 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:09:34.340 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:09:34.340 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:09:59.808 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:09:59.897 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:05.321 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:05.430 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:10:07.354 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:07.645 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:10:07.878 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-30T07:10:08.473 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:36 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:10:08.634 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.4 M 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-30T07:10:08.635 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:10:08.636 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:08.636 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.4 M 2024-03-30T07:10:08.636 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 6.2 M 2024-03-30T07:10:08.636 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:10:09.107 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-10 3.0 MB/s | 1.4 MB 00:00 2024-03-30T07:10:09.107 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:09.107 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.0 MB/s | 1.4 MB 00:00 2024-03-30T07:10:09.108 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:10:09.112 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:10:09.112 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:10:09.155 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:10:09.155 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:10:09.417 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:10:09.508 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:10:09.949 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:10:10.191 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1055. 1/1 2024-03-30T07:10:10.192 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:10.192 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:10:10.192 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:10.192 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:10.192 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:10:10.371 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-30T07:10:10.942 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:38 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:10:11.099 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 162 M 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 162 M 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 679 M 2024-03-30T07:10:11.100 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:10:12.735 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:12.996 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:13.227 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install ceph-fuse-debuginfo 2024-03-30T07:10:13.802 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:25 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:13.960 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:13.960 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.8 M 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.8 M 2024-03-30T07:10:13.961 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 28 M 2024-03-30T07:10:13.962 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:15.639 INFO:teuthology.orchestra.run.smithi047.stdout:ceph-fuse-debuginfo-18.2.2-1055.g55c4b44e.el9.x 3.5 MB/s | 5.8 MB 00:01 2024-03-30T07:10:15.640 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:15.640 INFO:teuthology.orchestra.run.smithi047.stdout:Total 3.4 MB/s | 5.8 MB 00:01 2024-03-30T07:10:15.640 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:15.644 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:15.644 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:15.718 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:15.722 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:16.483 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:16.581 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:17.142 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:17.403 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:17.404 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:17.404 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:17.404 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:17.404 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:17.404 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:17.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados-devel-debuginfo 2024-03-30T07:10:18.164 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:30 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:18.320 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 109 k 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:18.321 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:18.322 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 109 k 2024-03-30T07:10:18.322 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 440 k 2024-03-30T07:10:18.322 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:18.434 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-radosgw-debuginfo-18.2.2-1055.g55c4b44e.el 22 MB/s | 162 MB 00:07 2024-03-30T07:10:18.435 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:18.436 INFO:teuthology.orchestra.run.smithi052.stdout:Total 22 MB/s | 162 MB 00:07 2024-03-30T07:10:18.436 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:10:18.441 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:10:18.441 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:10:18.534 INFO:teuthology.orchestra.run.smithi047.stdout:librados-devel-debuginfo-18.2.2-1055.g55c4b44e. 513 kB/s | 109 kB 00:00 2024-03-30T07:10:18.534 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:18.534 INFO:teuthology.orchestra.run.smithi047.stdout:Total 509 kB/s | 109 kB 00:00 2024-03-30T07:10:18.535 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:18.538 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:18.538 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:18.574 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:18.574 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:18.697 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:18.793 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:19.161 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout: librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:19.428 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:19.469 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:10:19.470 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:10:19.609 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install libcephfs2-debuginfo 2024-03-30T07:10:20.181 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:32 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:20.336 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.1 M 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:20.337 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:20.338 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 5.1 M 2024-03-30T07:10:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 24 M 2024-03-30T07:10:20.338 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:20.795 INFO:teuthology.orchestra.run.smithi047.stdout:libcephfs2-debuginfo-18.2.2-1055.g55c4b44e.el9. 11 MB/s | 5.1 MB 00:00 2024-03-30T07:10:20.796 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:20.796 INFO:teuthology.orchestra.run.smithi047.stdout:Total 11 MB/s | 5.1 MB 00:00 2024-03-30T07:10:20.796 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:20.800 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:20.800 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:20.866 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:20.867 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:21.540 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:21.647 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:10:22.173 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:22.445 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:22.637 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librados2-debuginfo 2024-03-30T07:10:23.207 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:35 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:23.362 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:23.363 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:23.363 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:23.363 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:23.363 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 45 M 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 45 M 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 256 M 2024-03-30T07:10:23.364 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:25.556 INFO:teuthology.orchestra.run.smithi047.stdout:librados2-debuginfo-18.2.2-1055.g55c4b44e.el9.x 21 MB/s | 45 MB 00:02 2024-03-30T07:10:25.557 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:25.558 INFO:teuthology.orchestra.run.smithi047.stdout:Total 21 MB/s | 45 MB 00:02 2024-03-30T07:10:25.558 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:25.562 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:25.562 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:25.881 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:25.881 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:31.342 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:31.448 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:33.264 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout: librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:33.517 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:33.715 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install librbd1-debuginfo 2024-03-30T07:10:34.290 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:46 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:34.447 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:34.447 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 50 M 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 50 M 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 291 M 2024-03-30T07:10:34.448 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:37.011 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:10:37.042 INFO:teuthology.orchestra.run.smithi047.stdout:librbd1-debuginfo-18.2.2-1055.g55c4b44e.el9.x86 19 MB/s | 50 MB 00:02 2024-03-30T07:10:37.043 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:37.043 INFO:teuthology.orchestra.run.smithi047.stdout:Total 19 MB/s | 50 MB 00:02 2024-03-30T07:10:37.044 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:37.048 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:37.048 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:37.101 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:10:37.399 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:37.399 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:41.394 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x 1/1 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:41.648 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:10:41.850 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test-debuginfo 2024-03-30T07:10:42.422 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:10 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:10:42.579 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 582 M 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 582 M 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.6 G 2024-03-30T07:10:42.580 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:10:43.507 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:43.602 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:10:45.635 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout: librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:45.891 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:46.089 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-cephfs-debuginfo 2024-03-30T07:10:46.660 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:07:58 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 615 k 2024-03-30T07:10:46.815 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 615 k 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.1 M 2024-03-30T07:10:46.816 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:47.712 INFO:teuthology.orchestra.run.smithi047.stdout:python3-cephfs-debuginfo-18.2.2-1055.g55c4b44e. 687 kB/s | 615 kB 00:00 2024-03-30T07:10:47.712 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:47.713 INFO:teuthology.orchestra.run.smithi047.stdout:Total 685 kB/s | 615 kB 00:00 2024-03-30T07:10:47.713 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:47.717 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:47.717 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:47.761 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:47.762 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:47.940 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:48.047 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:48.578 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:48.841 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:10:48.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:48.842 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:48.842 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:48.842 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:48.842 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:49.080 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rados-debuginfo 2024-03-30T07:10:49.650 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:01 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:49.806 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 769 k 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 769 k 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.7 M 2024-03-30T07:10:49.807 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:50.515 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rados-debuginfo-18.2.2-1055.g55c4b44e.e 1.1 MB/s | 769 kB 00:00 2024-03-30T07:10:50.515 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:50.516 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.1 MB/s | 769 kB 00:00 2024-03-30T07:10:50.516 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:50.519 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:50.520 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:50.561 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:50.562 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:50.764 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:50.876 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:10:51.288 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:51.546 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:51.762 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rbd-debuginfo 2024-03-30T07:10:52.338 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:04 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:52.494 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:52.494 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.1 M 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.1 M 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 4.3 M 2024-03-30T07:10:52.495 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:53.460 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rbd-debuginfo-18.2.2-1055.g55c4b44e.el9 1.2 MB/s | 1.1 MB 00:00 2024-03-30T07:10:53.460 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:53.460 INFO:teuthology.orchestra.run.smithi047.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2024-03-30T07:10:53.461 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:53.464 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:53.464 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:53.509 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:53.509 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:53.738 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:53.853 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:54.370 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:54.629 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:54.630 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:54.630 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:54.630 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:54.630 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:54.630 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:54.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install python3-rgw-debuginfo 2024-03-30T07:10:55.540 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:07 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:55.695 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 337 k 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:55.696 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 337 k 2024-03-30T07:10:55.697 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 1.0 M 2024-03-30T07:10:55.697 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:56.413 INFO:teuthology.orchestra.run.smithi047.stdout:python3-rgw-debuginfo-18.2.2-1055.g55c4b44e.el9 471 kB/s | 337 kB 00:00 2024-03-30T07:10:56.414 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:56.414 INFO:teuthology.orchestra.run.smithi047.stdout:Total 470 kB/s | 337 kB 00:00 2024-03-30T07:10:56.414 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:56.418 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:56.418 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:56.458 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:56.458 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:56.601 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:56.788 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:57.152 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:57.412 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:10:57.627 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-fuse-debuginfo 2024-03-30T07:10:58.203 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:10 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:10:58.360 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:10:58.360 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:58.360 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 511 k 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 511 k 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 2.5 M 2024-03-30T07:10:58.361 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:10:59.032 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-fuse-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 762 kB/s | 511 kB 00:00 2024-03-30T07:10:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:10:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:Total 760 kB/s | 511 kB 00:00 2024-03-30T07:10:59.033 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:10:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:10:59.037 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:10:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:10:59.076 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:10:59.242 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:10:59.346 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:10:59.748 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:00.004 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:11:00.213 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-mirror-debuginfo 2024-03-30T07:11:00.789 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:12 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:11:00.945 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 50 M 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:00.946 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 50 M 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 281 M 2024-03-30T07:11:00.947 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:11:06.294 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-mirror-debuginfo-18.2.2-1055.g55c4b44e.el9. 9.4 MB/s | 50 MB 00:05 2024-03-30T07:11:06.295 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:11:06.295 INFO:teuthology.orchestra.run.smithi047.stdout:Total 9.4 MB/s | 50 MB 00:05 2024-03-30T07:11:06.295 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:11:06.302 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:11:06.302 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:11:06.644 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:11:06.645 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:11:08.017 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-test-debuginfo-18.2.2-1055.g55c4b44e.el9.x 23 MB/s | 582 MB 00:25 2024-03-30T07:11:08.017 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:11:08.018 INFO:teuthology.orchestra.run.smithi052.stdout:Total 23 MB/s | 582 MB 00:25 2024-03-30T07:11:08.018 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:11:08.024 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:11:08.024 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:11:11.627 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:11:11.627 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:11:12.670 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:11:12.779 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:11:14.731 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:11:14.996 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:11:14.997 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:14.997 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:11:14.997 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:11:14.997 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:14.997 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:11:15.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo yum -y install rbd-nbd-debuginfo 2024-03-30T07:11:15.774 INFO:teuthology.orchestra.run.smithi047.stdout:Last metadata expiration check: 0:08:27 ago on Sat 30 Mar 2024 07:02:48 AM UTC. 2024-03-30T07:11:15.930 INFO:teuthology.orchestra.run.smithi047.stdout:Dependencies resolved. 2024-03-30T07:11:15.931 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:15.931 INFO:teuthology.orchestra.run.smithi047.stdout: Package Arch Version Repo Size 2024-03-30T07:11:15.931 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:15.931 INFO:teuthology.orchestra.run.smithi047.stdout:Installing: 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.3 M 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction Summary 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:================================================================================ 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:Install 1 Package 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:Total download size: 1.3 M 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:Installed size: 5.5 M 2024-03-30T07:11:15.932 INFO:teuthology.orchestra.run.smithi047.stdout:Downloading Packages: 2024-03-30T07:11:16.396 INFO:teuthology.orchestra.run.smithi047.stdout:rbd-nbd-debuginfo-18.2.2-1055.g55c4b44e.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-03-30T07:11:16.396 INFO:teuthology.orchestra.run.smithi047.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:11:16.397 INFO:teuthology.orchestra.run.smithi047.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-03-30T07:11:16.397 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction check 2024-03-30T07:11:16.400 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction check succeeded. 2024-03-30T07:11:16.400 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction test 2024-03-30T07:11:16.443 INFO:teuthology.orchestra.run.smithi047.stdout:Transaction test succeeded. 2024-03-30T07:11:16.443 INFO:teuthology.orchestra.run.smithi047.stdout:Running transaction 2024-03-30T07:11:16.694 INFO:teuthology.orchestra.run.smithi047.stdout: Preparing : 1/1 2024-03-30T07:11:16.797 INFO:teuthology.orchestra.run.smithi047.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:11:17.185 INFO:teuthology.orchestra.run.smithi047.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:11:17.442 INFO:teuthology.orchestra.run.smithi047.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:11:17.443 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:17.443 INFO:teuthology.orchestra.run.smithi047.stdout:Installed: 2024-03-30T07:11:17.443 INFO:teuthology.orchestra.run.smithi047.stdout: rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:11:17.443 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:11:17.443 INFO:teuthology.orchestra.run.smithi047.stdout:Complete! 2024-03-30T07:11:17.637 DEBUG:teuthology.parallel:result is None 2024-03-30T07:12:16.239 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:12:16.880 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:12:24.710 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:24.970 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:12:25.226 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2024-03-30T07:12:25.796 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:53 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:12:25.897 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 is already installed. 2024-03-30T07:12:25.952 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:12:25.953 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-30T07:12:25.953 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:12:26.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-debuginfo 2024-03-30T07:12:26.607 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:54 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:12:26.764 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:12:26.765 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:26.765 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:12:26.765 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:26.765 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 19 M 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 19 M 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 101 M 2024-03-30T07:12:26.766 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:12:29.072 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mgr-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 8.3 MB/s | 19 MB 00:02 2024-03-30T07:12:29.072 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:12:29.073 INFO:teuthology.orchestra.run.smithi052.stdout:Total 8.3 MB/s | 19 MB 00:02 2024-03-30T07:12:29.073 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:12:29.077 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:12:29.077 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:12:29.228 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:12:29.229 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:12:31.551 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:12:31.656 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:32.582 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:32.848 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:32.849 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:32.849 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:12:32.849 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:12:32.849 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:32.849 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:12:33.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mds-debuginfo 2024-03-30T07:12:33.592 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:01 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:12:33.747 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:12:33.747 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:33.747 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 47 M 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 47 M 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 171 M 2024-03-30T07:12:33.748 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:12:36.332 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mds-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 18 MB/s | 47 MB 00:02 2024-03-30T07:12:36.333 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:12:36.333 INFO:teuthology.orchestra.run.smithi052.stdout:Total 18 MB/s | 47 MB 00:02 2024-03-30T07:12:36.334 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:12:36.338 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:12:36.338 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:12:36.665 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:12:36.665 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:12:41.643 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:12:41.742 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:43.134 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:43.391 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:12:43.571 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mon-debuginfo 2024-03-30T07:12:44.141 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:12 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:12:44.296 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 98 M 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:44.297 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 98 M 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 450 M 2024-03-30T07:12:44.298 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:12:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mon-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 22 MB/s | 98 MB 00:04 2024-03-30T07:12:48.785 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:12:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:Total 22 MB/s | 98 MB 00:04 2024-03-30T07:12:48.786 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:12:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:12:48.790 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:12:49.427 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:12:49.428 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:13:00.357 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:13:00.460 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:03.380 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:03.647 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:03.647 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:03.647 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:13:03.647 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:13:03.647 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:03.648 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:13:03.825 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-osd-debuginfo 2024-03-30T07:13:04.397 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:32 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:13:04.555 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 286 M 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:13:04.556 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:13:04.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:04.557 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 286 M 2024-03-30T07:13:04.557 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.2 G 2024-03-30T07:13:04.557 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:13:18.744 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-osd-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 20 MB/s | 286 MB 00:14 2024-03-30T07:13:18.744 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:13:18.745 INFO:teuthology.orchestra.run.smithi052.stdout:Total 20 MB/s | 286 MB 00:14 2024-03-30T07:13:18.745 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:13:18.749 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:13:18.749 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:13:20.535 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:13:20.535 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:13:51.523 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:13:51.617 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:58.953 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:59.214 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:13:59.215 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:59.215 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:13:59.215 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:13:59.215 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:13:59.215 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:13:59.441 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse-debuginfo 2024-03-30T07:14:00.015 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:28 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:00.169 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:00.170 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:00.170 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:00.170 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:00.170 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:00.170 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.8 M 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.8 M 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 28 M 2024-03-30T07:14:00.171 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:00.677 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-fuse-debuginfo-18.2.2-1055.g55c4b44e.el9.x 11 MB/s | 5.8 MB 00:00 2024-03-30T07:14:00.677 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:00.678 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 5.8 MB 00:00 2024-03-30T07:14:00.678 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:00.681 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:00.681 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:00.752 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:00.752 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:01.513 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:02.210 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:02.472 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:02.650 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel-debuginfo 2024-03-30T07:14:03.221 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:31 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:03.376 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:03.377 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:03.377 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:03.377 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:03.377 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:03.377 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 109 k 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 109 k 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 440 k 2024-03-30T07:14:03.378 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:03.587 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-debuginfo-18.2.2-1055.g55c4b44e. 518 kB/s | 109 kB 00:00 2024-03-30T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:Total 515 kB/s | 109 kB 00:00 2024-03-30T07:14:03.588 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:03.592 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:03.592 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:03.627 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:03.627 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:03.742 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:03.854 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:04.245 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:04.503 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:04.503 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:04.503 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:04.503 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:04.504 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:04.504 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:04.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2-debuginfo 2024-03-30T07:14:05.250 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:33 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:05.405 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 5.1 M 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:05.406 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.1 M 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 24 M 2024-03-30T07:14:05.407 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:06.357 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs2-debuginfo-18.2.2-1055.g55c4b44e.el9. 5.4 MB/s | 5.1 MB 00:00 2024-03-30T07:14:06.358 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:06.359 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.4 MB/s | 5.1 MB 00:00 2024-03-30T07:14:06.359 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:06.362 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:06.362 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:06.431 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:06.431 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:07.217 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:07.784 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:08.057 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:08.057 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:08.057 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:08.057 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:08.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:08.058 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:08.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2-debuginfo 2024-03-30T07:14:08.828 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:36 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:08.984 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 45 M 2024-03-30T07:14:08.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 45 M 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 256 M 2024-03-30T07:14:08.986 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:13.476 INFO:teuthology.orchestra.run.smithi052.stdout:librados2-debuginfo-18.2.2-1055.g55c4b44e.el9.x 10 MB/s | 45 MB 00:04 2024-03-30T07:14:13.477 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:13.477 INFO:teuthology.orchestra.run.smithi052.stdout:Total 10 MB/s | 45 MB 00:04 2024-03-30T07:14:13.478 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:13.481 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:13.481 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:13.800 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:13.800 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:19.240 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:19.344 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:21.177 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_ 1/1 2024-03-30T07:14:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:21.429 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:21.429 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:21.430 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:21.430 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:21.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1-debuginfo 2024-03-30T07:14:22.184 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:07:50 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:22.340 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:22.341 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:22.341 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:22.341 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:22.341 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 50 M 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 50 M 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 291 M 2024-03-30T07:14:22.342 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:24.745 INFO:teuthology.orchestra.run.smithi052.stdout:librbd1-debuginfo-18.2.2-1055.g55c4b44e.el9.x86 21 MB/s | 50 MB 00:02 2024-03-30T07:14:24.746 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:24.746 INFO:teuthology.orchestra.run.smithi052.stdout:Total 21 MB/s | 50 MB 00:02 2024-03-30T07:14:24.747 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:24.751 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:24.751 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:25.101 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:25.101 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:31.231 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:31.338 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:33.427 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:33.685 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:33.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs-debuginfo 2024-03-30T07:14:34.460 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:02 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 615 k 2024-03-30T07:14:34.618 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 615 k 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.1 M 2024-03-30T07:14:34.619 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:34.873 INFO:teuthology.orchestra.run.smithi052.stdout:python3-cephfs-debuginfo-18.2.2-1055.g55c4b44e. 2.4 MB/s | 615 kB 00:00 2024-03-30T07:14:34.874 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:34.874 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.4 MB/s | 615 kB 00:00 2024-03-30T07:14:34.874 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:34.877 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:34.878 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:34.917 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:34.918 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:35.124 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:35.224 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:35.622 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9 1/1 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:35.876 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:36.064 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados-debuginfo 2024-03-30T07:14:36.636 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:04 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:36.791 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 769 k 2024-03-30T07:14:36.792 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 769 k 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2024-03-30T07:14:36.793 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rados-debuginfo-18.2.2-1055.g55c4b44e.e 1.7 MB/s | 769 kB 00:00 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.7 MB/s | 769 kB 00:00 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:37.310 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:37.466 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:37.569 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:14:37.968 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9. 1/1 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:38.225 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:38.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd-debuginfo 2024-03-30T07:14:39.038 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:07 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:39.194 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.1 M 2024-03-30T07:14:39.195 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.1 M 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 4.3 M 2024-03-30T07:14:39.196 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:39.490 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rbd-debuginfo-18.2.2-1055.g55c4b44e.el9 3.9 MB/s | 1.1 MB 00:00 2024-03-30T07:14:39.490 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:39.491 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2024-03-30T07:14:39.491 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:39.491 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:39.491 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:39.533 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:39.534 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:39.865 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:40.526 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:40.526 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:40.744 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw-debuginfo 2024-03-30T07:14:41.317 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:09 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:41.473 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:41.473 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:41.473 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 337 k 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 337 k 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.0 M 2024-03-30T07:14:41.474 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:41.840 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rgw-debuginfo-18.2.2-1055.g55c4b44e.el9 923 kB/s | 337 kB 00:00 2024-03-30T07:14:41.840 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:41.841 INFO:teuthology.orchestra.run.smithi052.stdout:Total 919 kB/s | 337 kB 00:00 2024-03-30T07:14:41.841 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:41.844 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:41.845 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:41.882 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:42.022 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:42.134 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:42.522 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:42.776 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x8 1/1 2024-03-30T07:14:42.777 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:42.777 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:42.777 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:42.777 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:42.777 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:42.991 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse-debuginfo 2024-03-30T07:14:43.564 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:11 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:43.723 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:43.723 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:43.723 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:43.723 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 511 k 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 511 k 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.5 M 2024-03-30T07:14:43.724 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:44.001 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-debuginfo-18.2.2-1055.g55c4b44e.el9.x8 1.8 MB/s | 511 kB 00:00 2024-03-30T07:14:44.002 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:44.002 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.8 MB/s | 511 kB 00:00 2024-03-30T07:14:44.002 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:44.006 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:44.006 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:44.043 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:44.044 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:44.210 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:44.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:14:44.693 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_6 1/1 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:44.950 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:45.144 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror-debuginfo 2024-03-30T07:14:45.718 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:13 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:45.874 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 50 M 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:45.875 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 50 M 2024-03-30T07:14:45.876 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 281 M 2024-03-30T07:14:45.876 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:48.438 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-debuginfo-18.2.2-1055.g55c4b44e.el9. 20 MB/s | 50 MB 00:02 2024-03-30T07:14:48.439 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:48.439 INFO:teuthology.orchestra.run.smithi052.stdout:Total 20 MB/s | 50 MB 00:02 2024-03-30T07:14:48.439 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:48.444 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:48.444 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:48.787 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:54.817 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:54.914 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:56.875 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:57.133 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86 1/1 2024-03-30T07:14:57.133 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:57.133 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:57.134 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:57.134 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:57.134 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:57.328 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd-debuginfo 2024-03-30T07:14:57.899 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:08:25 ago on Sat 30 Mar 2024 07:06:32 AM UTC. 2024-03-30T07:14:58.055 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-30T07:14:58.056 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:58.056 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-30T07:14:58.056 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1055.g55c4b44e.el9 ceph 1.3 M 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.3 M 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 5.5 M 2024-03-30T07:14:58.057 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-30T07:14:58.531 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-debuginfo-18.2.2-1055.g55c4b44e.el9.x86 2.7 MB/s | 1.3 MB 00:00 2024-03-30T07:14:58.532 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-30T07:14:58.532 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-03-30T07:14:58.532 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-30T07:14:58.535 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-30T07:14:58.535 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-30T07:14:58.578 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-30T07:14:58.578 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-30T07:14:58.826 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-30T07:14:58.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:59.387 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:59.639 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 1/1 2024-03-30T07:14:59.639 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:59.640 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-30T07:14:59.640 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo-2:18.2.2-1055.g55c4b44e.el9.x86_64 2024-03-30T07:14:59.640 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:14:59.640 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-30T07:14:59.838 DEBUG:teuthology.parallel:result is None 2024-03-30T07:14:59.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:15:00.056 DEBUG:teuthology.orchestra.run.smithi047:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T07:15:00.081 INFO:teuthology.orchestra.run.smithi047.stdout:18.2.2-1055.g55c4b44e.el9 2024-03-30T07:15:00.082 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1055.g55c4b44e.el9 2024-03-30T07:15:00.082 INFO:teuthology.task.install:The correct ceph version 18.2.2-1055.g55c4b44e is installed. 2024-03-30T07:15:00.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:15:00.227 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T07:15:00.248 INFO:teuthology.orchestra.run.smithi052.stdout:18.2.2-1055.g55c4b44e.el9 2024-03-30T07:15:00.249 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1055.g55c4b44e.el9 2024-03-30T07:15:00.249 INFO:teuthology.task.install:The correct ceph version 18.2.2-1055.g55c4b44e is installed. 2024-03-30T07:15:00.250 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T07:15:00.251 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:00.251 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T07:15:00.284 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:00.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T07:15:00.316 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T07:15:00.316 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:00.316 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T07:15:00.354 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T07:15:00.424 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:00.424 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T07:15:00.452 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T07:15:00.519 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T07:15:00.519 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:00.520 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T07:15:00.549 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T07:15:00.618 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:00.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T07:15:00.646 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T07:15:00.713 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T07:15:00.713 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:00.713 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T07:15:00.743 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T07:15:00.813 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:00.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T07:15:00.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T07:15:00.907 INFO:teuthology.run_tasks:Running task ceph... 2024-03-30T07:15:00.973 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-30T07:15:00.974 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-03-30T07:15:00.976 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2024-03-30T07:15:01.003 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-30T07:15:01.004 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T07:15:01.046 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T07:15:01.073 INFO:tasks.ceph:Creating extra log directories... 2024-03-30T07:15:01.073 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T07:15:01.116 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T07:15:01.156 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-30T07:15:01.156 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-30T07:15:01.157 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630', 'branch': 'wip-yuri11-testing-2024-03-28-0753-reef', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7629630', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2024-03-28-0753-reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri11-testing-2024-03-28-0753-reef', 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4127, 'sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri11-testing-2024-03-28-0753-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCp1NYDh5mND57I3ETURfiQ7fqoDP3o6d3ZlghAYPMrkU0/MSYTdfBhjOBfSrbPKToN6CgqW5VCkIWDPzd1CX38=', 'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMfXVMcbopoc/81qCl9RxIlD5evhICEsqNOexv19l/QmBJ4aBrFJZwxWdExXnqpSLRdVORGZlHFF2f8UAo1Zr0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-29_19:10:12', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-30T07:15:01.157 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T07:15:01.185 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T07:15:01.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T07:15:01.242 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T07:15:01.269 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:01.269 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T07:15:01.324 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T07:15:01.324 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-03-30T07:15:01.379 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-30 07:04:32.427498980 +0000 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-30 07:02:03.293530101 +0000 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-30 07:02:03.293530101 +0000 2024-03-30T07:15:01.380 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-30 07:02:03.293530101 +0000 2024-03-30T07:15:01.380 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T07:15:01.446 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-30T07:15:01.447 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-30T07:15:01.447 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.00010007 s, 5.1 MB/s 2024-03-30T07:15:01.448 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T07:15:01.504 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-30 07:04:32.427498980 +0000 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-30 07:02:03.641522512 +0000 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-30 07:02:03.641522512 +0000 2024-03-30T07:15:01.561 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-30 07:02:03.641522512 +0000 2024-03-30T07:15:01.562 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T07:15:01.628 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-30T07:15:01.628 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-30T07:15:01.628 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000100798 s, 5.1 MB/s 2024-03-30T07:15:01.629 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T07:15:01.685 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-03-30T07:15:01.740 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T07:15:01.740 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:01.740 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-30 07:04:32.427498980 +0000 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-30 07:02:03.983515055 +0000 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-30 07:02:03.983515055 +0000 2024-03-30T07:15:01.741 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-30 07:02:03.983515055 +0000 2024-03-30T07:15:01.741 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T07:15:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-30T07:15:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-30T07:15:01.806 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.1944e-05 s, 5.6 MB/s 2024-03-30T07:15:01.807 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T07:15:01.862 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-03-30 07:04:32.428498958 +0000 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-03-30 07:02:04.322507663 +0000 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-03-30 07:02:04.322507663 +0000 2024-03-30T07:15:01.917 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: 2024-03-30 07:02:04.322507663 +0000 2024-03-30T07:15:01.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T07:15:01.982 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-03-30T07:15:01.983 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-03-30T07:15:01.983 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 9.4476e-05 s, 5.4 MB/s 2024-03-30T07:15:01.984 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T07:15:02.040 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:02.040 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:02.040 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T07:15:02.058 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T07:15:02.059 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:02.116 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-30 07:08:24.063546652 +0000 2024-03-30T07:15:02.117 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-30 07:01:05.402774759 +0000 2024-03-30T07:15:02.117 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-30 07:01:05.402774759 +0000 2024-03-30T07:15:02.117 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-30 07:01:05.402774759 +0000 2024-03-30T07:15:02.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T07:15:02.187 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-30T07:15:02.187 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-30T07:15:02.187 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000102989 s, 5.0 MB/s 2024-03-30T07:15:02.188 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T07:15:02.247 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-03-30T07:15:02.303 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-30 07:08:24.064546638 +0000 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-30 07:01:05.744769988 +0000 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-30 07:01:05.744769988 +0000 2024-03-30T07:15:02.304 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-30 07:01:05.744769988 +0000 2024-03-30T07:15:02.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T07:15:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-30T07:15:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-30T07:15:02.372 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00010679 s, 4.8 MB/s 2024-03-30T07:15:02.373 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T07:15:02.431 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-30 07:08:24.064546638 +0000 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-30 07:01:06.074765384 +0000 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-30 07:01:06.074765384 +0000 2024-03-30T07:15:02.487 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-30 07:01:06.074765384 +0000 2024-03-30T07:15:02.488 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T07:15:02.554 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-30T07:15:02.555 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-30T07:15:02.555 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000104058 s, 4.9 MB/s 2024-03-30T07:15:02.556 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T07:15:02.612 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-03-30T07:15:02.668 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T07:15:02.668 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T07:15:02.668 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-30T07:15:02.668 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T07:15:02.669 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T07:15:02.669 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-30 07:08:24.064546638 +0000 2024-03-30T07:15:02.669 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-30 07:01:06.408760725 +0000 2024-03-30T07:15:02.669 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-30 07:01:06.408760725 +0000 2024-03-30T07:15:02.669 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-30 07:01:06.408760725 +0000 2024-03-30T07:15:02.669 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T07:15:02.734 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-30T07:15:02.735 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-30T07:15:02.735 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 8.876e-05 s, 5.8 MB/s 2024-03-30T07:15:02.736 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T07:15:02.792 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:02.792 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T07:15:02.792 INFO:tasks.ceph:Generating config... 2024-03-30T07:15:02.793 INFO:tasks.ceph:[client] debug monc = 20 2024-03-30T07:15:02.793 INFO:tasks.ceph:[client] debug ms = 1 2024-03-30T07:15:02.793 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-30T07:15:02.793 INFO:tasks.ceph:[client] debug rados = 20 2024-03-30T07:15:02.793 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-30T07:15:02.793 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-30T07:15:02.794 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-30T07:15:02.794 INFO:tasks.ceph:[global] ms type = async 2024-03-30T07:15:02.794 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-30T07:15:02.794 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-30T07:15:02.795 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-30T07:15:02.795 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-30T07:15:02.795 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-30T07:15:02.796 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-30T07:15:02.797 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-30T07:15:02.797 INFO:tasks.ceph:Setting up mon.a... 2024-03-30T07:15:02.797 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-30T07:15:02.854 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.keyring 2024-03-30T07:15:02.857 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-30T07:15:02.910 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T07:15:02.985 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.47'), ('mon.c', '[v2:172.21.15.47:3301,v1:172.21.15.47:6790]'), ('mon.b', '172.21.15.52')] 2024-03-30T07:15:02.986 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790],172.21.15.52', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-30T07:15:02.987 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:02.987 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T07:15:03.043 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.47 --addv c '[v2:172.21.15.47:3301,v1:172.21.15.47:6790]' --add b 172.21.15.52 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:03.133 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:03.133 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:03.133 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = pacific 2024-03-30T07:15:03.133 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 16 (pacific) 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 1 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:03.134 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-30T07:15:03.135 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T07:15:03.192 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bc5c1745-fe46-4846-b9b2-29d88e4530d9... 2024-03-30T07:15:03.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T07:15:03.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T07:15:03.284 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-03-30T07:15:03.284 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-03-30T07:15:03.284 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T07:15:03.284 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-03-30T07:15:03.284 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-03-30T07:15:03.285 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-03-30T07:15:03.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.286 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-03-30T07:15:03.286 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-03-30T07:15:03.286 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.286 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-03-30T07:15:03.287 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.288 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790],172.21.15.52" 2024-03-30T07:15:03.289 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: mon election default strategy = 1 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 5000 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: ms type = async 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: osd heartbeat grace = 80 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-03-30T07:15:03.290 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-03-30T07:15:03.291 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-03-30T07:15:03.292 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bdev async discard = True 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bdev enable discard = True 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore allocator = bitmap 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore zero block detection = True 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 1 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 1 2024-03-30T07:15:03.293 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 10 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-03-30T07:15:03.294 INFO:teuthology.orchestra.run.smithi047.stdout: osd class default list = * 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd class load list = * 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd client watch timeout = 120 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify cached snaps = True 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify missing on start = True 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd fast shutdown = False 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock override recovery settings = True 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue = debug_random 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue cut off = debug_random 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: osd shutdown pgref assert = True 2024-03-30T07:15:03.295 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-30T07:15:03.296 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-03-30T07:15:03.297 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T07:15:03.298 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon min osdmap epochs = 50 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd crush smoke test = False 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune interval = 2 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune min = 15 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune txsize = 2 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub interval = 300 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-03-30T07:15:03.299 INFO:teuthology.orchestra.run.smithi047.stdout: paxos service trim min = 10 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T07:15:03.300 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout: debug objecter = 20 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout: debug rados = 20 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-03-30T07:15:03.301 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-03-30T07:15:03.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-03-30T07:15:03.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-03-30T07:15:03.304 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = "172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790],172.21.15.52" 2024-03-30T07:15:03.305 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: mon election default strategy = 1 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: ms type = async 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: osd heartbeat grace = 80 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-03-30T07:15:03.306 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-03-30T07:15:03.307 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-03-30T07:15:03.307 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-03-30T07:15:03.308 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: bdev async discard = True 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: bdev enable discard = True 2024-03-30T07:15:03.309 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore allocator = bitmap 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore zero block detection = True 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 1 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 1 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 10 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2024-03-30T07:15:03.310 INFO:teuthology.orchestra.run.smithi052.stdout: osd class default list = * 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd class load list = * 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd client watch timeout = 120 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify cached snaps = True 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify missing on start = True 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd fast shutdown = False 2024-03-30T07:15:03.311 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock override recovery settings = True 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue = debug_random 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue cut off = debug_random 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: osd shutdown pgref assert = True 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-30T07:15:03.312 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.313 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-03-30T07:15:03.314 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon min osdmap epochs = 50 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd crush smoke test = False 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune interval = 2 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune min = 15 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune txsize = 2 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon scrub interval = 300 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: paxos service trim min = 10 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-30T07:15:03.315 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: debug objecter = 20 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout: debug rados = 20 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.c] 2024-03-30T07:15:03.316 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.b] 2024-03-30T07:15:03.318 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-30T07:15:03.318 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-30T07:15:03.412 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-30T07:15:03.412 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:03.412 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-30T07:15:03.427 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:03.427 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-30T07:15:03.481 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:15:03.482 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:03.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T07:15:03.482 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T07:15:03.560 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:03.560 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:03.616 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:15:03.617 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:03.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T07:15:03.617 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T07:15:03.663 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:03.663 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:03.720 INFO:tasks.ceph:Setting up mon nodes... 2024-03-30T07:15:03.720 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-30T07:15:03.720 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T07:15:03.774 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T07:15:03.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T07:15:03.843 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T07:15:03.846 INFO:tasks.ceph:Setting up mds nodes... 2024-03-30T07:15:03.846 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-30T07:15:03.846 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T07:15:03.888 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-30T07:15:03.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T07:15:03.949 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-30T07:15:03.966 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-30T07:15:03.966 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T07:15:03.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-30T07:15:03.993 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:03.993 INFO:tasks.ceph:role: osd.0 2024-03-30T07:15:03.993 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:15:03.993 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T07:15:04.059 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:04.059 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:04.060 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:04.092 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-30T07:15:04.094 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-30T07:15:04.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-30T07:15:04.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-30T07:15:04.236 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-30T07:15:04.302 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:04.302 INFO:tasks.ceph:role: osd.1 2024-03-30T07:15:04.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:15:04.302 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T07:15:04.368 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:04.368 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:04.368 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:04.369 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:04.400 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-30T07:15:04.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-30T07:15:04.402 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-30T07:15:04.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-30T07:15:04.549 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-30T07:15:04.616 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:04.616 INFO:tasks.ceph:role: osd.2 2024-03-30T07:15:04.616 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:15:04.616 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T07:15:04.682 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:04.682 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:04.682 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:04.682 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:04.682 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:04.683 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:04.683 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:04.683 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:04.683 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:04.683 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:04.714 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-30T07:15:04.716 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-30T07:15:04.716 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-30T07:15:04.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-30T07:15:04.857 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-30T07:15:04.924 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:04.924 INFO:tasks.ceph:role: osd.3 2024-03-30T07:15:04.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:15:04.925 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T07:15:04.990 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:04.990 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:04.990 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:04.990 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:04.990 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:04.991 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:04.991 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:04.991 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:04.991 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:04.991 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:05.022 INFO:teuthology.orchestra.run.smithi047.stdout:Discarding blocks...Done. 2024-03-30T07:15:05.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-03-30T07:15:05.024 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-30T07:15:05.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-30T07:15:05.165 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T07:15:05.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:05.255+0000 7f2ac16244c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-30T07:15:08.097 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T07:15:08.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-30T07:15:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-30T07:15:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T07:15:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T07:15:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T07:15:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:08.218+0000 7f710b6124c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-30T07:15:11.069 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T07:15:11.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.189+0000 7f5f139394c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.189+0000 7f5f139394c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.189+0000 7f5f139394c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.189+0000 7f5f139394c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.189+0000 7f5f139394c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T07:15:11.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:11.190+0000 7f5f139394c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-30T07:15:14.030 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T07:15:14.064 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.154+0000 7fa5b84194c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.154+0000 7fa5b84194c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.154+0000 7fa5b84194c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.154+0000 7fa5b84194c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.154+0000 7fa5b84194c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T07:15:14.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:14.155+0000 7fa5b84194c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-30T07:15:16.993 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T07:15:17.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-30T07:15:17.055 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:17.056 INFO:tasks.ceph:role: osd.4 2024-03-30T07:15:17.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:15:17.056 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T07:15:17.127 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:17.127 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:17.127 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:17.127 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:17.128 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:17.160 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-30T07:15:17.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-30T07:15:17.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-30T07:15:17.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-30T07:15:17.310 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-30T07:15:17.378 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:17.378 INFO:tasks.ceph:role: osd.5 2024-03-30T07:15:17.378 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:15:17.378 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:17.446 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:17.476 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-30T07:15:17.478 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-30T07:15:17.479 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-30T07:15:17.549 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-30T07:15:17.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-30T07:15:17.685 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:17.685 INFO:tasks.ceph:role: osd.6 2024-03-30T07:15:17.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:15:17.685 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:17.752 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:17.753 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:17.753 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:17.753 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:17.753 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:17.783 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-30T07:15:17.785 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-30T07:15:17.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-30T07:15:17.856 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-30T07:15:17.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-30T07:15:17.993 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T07:15:17.993 INFO:tasks.ceph:role: osd.7 2024-03-30T07:15:17.994 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:15:17.994 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T07:15:18.059 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T07:15:18.060 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T07:15:18.092 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-30T07:15:18.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-30T07:15:18.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-30T07:15:18.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-30T07:15:18.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.331+0000 7fb3aaab44c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-30T07:15:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.331+0000 7fb3aaab44c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-30T07:15:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.332+0000 7fb3aaab44c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T07:15:18.333 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.332+0000 7fb3aaab44c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T07:15:18.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.332+0000 7fb3aaab44c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T07:15:18.334 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:18.332+0000 7fb3aaab44c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-30T07:15:21.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T07:15:21.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T07:15:21.296 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:21.295+0000 7f7a0cec54c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-30T07:15:24.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T07:15:24.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:24.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-30T07:15:24.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-30T07:15:24.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T07:15:24.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T07:15:24.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T07:15:24.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:24.256+0000 7f0b1560f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-30T07:15:27.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T07:15:27.126 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:27.219 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-30T07:15:27.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-30T07:15:27.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T07:15:27.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T07:15:27.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T07:15:27.220 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:27.218+0000 7fee03c8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-30T07:15:30.065 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T07:15:30.094 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-30T07:15:30.094 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.094 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-30T07:15:30.126 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.126 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-30T07:15:30.197 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-30T07:15:30.268 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.268 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-30T07:15:30.338 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.338 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-30T07:15:30.408 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.408 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-30T07:15:30.439 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.439 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-30T07:15:30.508 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.508 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-30T07:15:30.576 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-30T07:15:30.645 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.645 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-30T07:15:30.713 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:30.714 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-30T07:15:30.731 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:30.732 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-30T07:15:30.769 INFO:tasks.ceph:Adding keys to all mons... 2024-03-30T07:15:30.769 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T07:15:30.774 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.y] 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD3uwdmXVQ0LhAA3Jm4i8EPDKbcAEZYsDkDVw== 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD5uwdmR4RTDxAAJ1IKH0CTnVWmIxAuoMsMHQ== 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD8uwdmY64bDRAAgY1lV7iz+3t4GQs2mpqTAA== 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD/uwdmSl9lCxAA9g196XSs1tt9mkMZJe+EPQ== 2024-03-30T07:15:30.812 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQACvAdmO/ZPCRAAMurlMj1BqSD2LI1hSN94Ag== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD3uwdmRulOMhAAoZpMruCLg4t8uOR2Z2YBSw== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAGvAdmF93YExAAO3xD6fXHgrBnLOKkMaS3hg== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAJvAdmUYupERAARaNUjKgq8hCSVSCtIR/Qyw== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAMvAdmp2FXDxAAglHN8RnFvo4guZv3pAVtSQ== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQAPvAdmHdMVDRAAXFoUBSH3NFT3COyiYmBRxA== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD3uwdm4Iv5NBAAJNAOofpLXZfqbeh2BuXSpg== 2024-03-30T07:15:30.813 INFO:teuthology.orchestra.run.smithi047.stdout:[client.1] 2024-03-30T07:15:30.814 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD3uwdm6l2kOBAAM0kVd8uGxayTn09onYf+Ng== 2024-03-30T07:15:30.835 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.y] 2024-03-30T07:15:30.835 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD3uwdmXVQ0LhAA3Jm4i8EPDKbcAEZYsDkDVw== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD5uwdmR4RTDxAAJ1IKH0CTnVWmIxAuoMsMHQ== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD8uwdmY64bDRAAgY1lV7iz+3t4GQs2mpqTAA== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD/uwdmSl9lCxAA9g196XSs1tt9mkMZJe+EPQ== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQACvAdmO/ZPCRAAMurlMj1BqSD2LI1hSN94Ag== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD3uwdmRulOMhAAoZpMruCLg4t8uOR2Z2YBSw== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.4] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAGvAdmF93YExAAO3xD6fXHgrBnLOKkMaS3hg== 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.5] 2024-03-30T07:15:30.836 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAJvAdmUYupERAARaNUjKgq8hCSVSCtIR/Qyw== 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.6] 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAMvAdmp2FXDxAAglHN8RnFvo4guZv3pAVtSQ== 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.7] 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAPvAdmHdMVDRAAXFoUBSH3NFT3COyiYmBRxA== 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD3uwdm4Iv5NBAAJNAOofpLXZfqbeh2BuXSpg== 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2024-03-30T07:15:30.837 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD3uwdm6l2kOBAAM0kVd8uGxayTn09onYf+Ng== 2024-03-30T07:15:30.838 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T07:15:30.858 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T07:15:30.931 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:30.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:30.986 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:30.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.042 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.093 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.146 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T07:15:31.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T07:15:31.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.201 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.252 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.254 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.305 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.308 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T07:15:31.412 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T07:15:31.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T07:15:31.468 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T07:15:31.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T07:15:31.524 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-30T07:15:31.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-30T07:15:31.550 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T07:15:31.646 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-30T07:15:31.673 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-30T07:15:31.742 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T07:15:31.838 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-30T07:15:31.864 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-30T07:15:31.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T07:15:31.988 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-30T07:15:32.014 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:32.016 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T07:15:32.071 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-30T07:15:32.072 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-30T07:15:32.072 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-30T07:15:32.072 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-30T07:15:32.074 INFO:tasks.ceph.mon.a:Started 2024-03-30T07:15:32.074 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-30T07:15:32.075 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-30T07:15:32.075 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-30T07:15:32.077 INFO:tasks.ceph.mon.c:Started 2024-03-30T07:15:32.077 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-30T07:15:32.077 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-30T07:15:32.077 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-30T07:15:32.114 INFO:tasks.ceph.mon.b:Started 2024-03-30T07:15:32.114 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-30T07:15:32.115 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-30T07:15:32.115 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-30T07:15:32.118 INFO:tasks.ceph.mgr.y:Started 2024-03-30T07:15:32.119 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-30T07:15:32.119 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-30T07:15:32.121 INFO:tasks.ceph.mgr.x:Started 2024-03-30T07:15:32.122 INFO:tasks.ceph:Setting crush tunables to default 2024-03-30T07:15:32.122 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd crush tunables default 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.222+0000 7f6c79ac4640 10 monclient: get_monmap_and_config 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.222+0000 7f6c79ac4640 10 monclient: build_initial_monmap 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.223+0000 7f6c79ac4640 10 monclient: monmap: 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:32.223 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:32.223712+0000 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:32.223712+0000 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:32.224 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:32.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.223+0000 7f6c79ac4640 1 Processor -- start 2024-03-30T07:15:32.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.223+0000 7f6c79ac4640 1 -- start start 2024-03-30T07:15:32.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.223+0000 7f6c79ac4640 10 monclient: init 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c740b4280 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c7413e930 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c7413eda0 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740aa3c0 con 0x7f6c7413e930 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c7413eda0 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740aa630 con 0x7f6c740b4280 2024-03-30T07:15:32.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c79ac4640 10 monclient(hunting): authenticate will time out at 1458.339546s 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c73fff640 1 -- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 msgr2=0x7f6c74141d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3300/0 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c73fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c72ffd640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c7413e930 legacy=0x7f6c74141670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c740a8990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:32.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.224+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:32.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.425+0000 7f6c73fff640 1 -- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 msgr2=0x7f6c74141d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3300/0 2024-03-30T07:15:32.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.425+0000 7f6c73fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:32.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.425+0000 7f6c72ffd640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c7413e930 legacy=0x7f6c74141670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:32.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.425+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c740a8990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:32.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.425+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:32.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.826+0000 7f6c73fff640 1 -- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 msgr2=0x7f6c74141d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3300/0 2024-03-30T07:15:32.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.826+0000 7f6c73fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:32.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.826+0000 7f6c72ffd640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c7413e930 legacy=0x7f6c74141670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:32.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.826+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c740a8990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:32.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:32.826+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:33.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:33.627+0000 7f6c73fff640 1 -- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 msgr2=0x7f6c74141d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3300/0 2024-03-30T07:15:33.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:33.627+0000 7f6c73fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:33.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:33.627+0000 7f6c72ffd640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c7413e930 legacy=0x7f6c74141670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:33.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:33.627+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c740a8990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:33.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:33.627+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:35.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): tick 2024-03-30T07:15:35.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): _check_auth_tickets 2024-03-30T07:15:35.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 monclient(hunting): continuing hunt 2024-03-30T07:15:35.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-30T07:15:35.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c740a8990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c740a8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 -- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 msgr2=0x7f6c74141d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c7413eda0 0x7f6c74141d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c7413e930 legacy=0x7f6c74141670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient: _add_conns ranks=[0,2] 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c600036b0 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-b con 0x7f6c600072d0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740a9210 con 0x7f6c600036b0 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c600072d0 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c717fa640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:35.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c737fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f6c600036b0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-30T07:15:35.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.224+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:35.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.425+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:35.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.425+0000 7f6c737fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f6c600036b0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-30T07:15:35.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.425+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:35.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.826+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:35.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.826+0000 7f6c737fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f6c600036b0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-30T07:15:35.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:35.826+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:36.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:36.627+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:36.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:36.627+0000 7f6c737fe640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f6c600036b0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-03-30T07:15:36.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:36.627+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): tick 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): _check_auth_tickets 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 monclient(hunting): continuing hunt 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f6c600036b0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c73fff640 1 -- reap_dead start 2024-03-30T07:15:38.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6c740b4280 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-b con 0x7f6c600072d0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c6000a3f0 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740aa3c0 con 0x7f6c6000a3f0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c740b4280 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740aa630 con 0x7f6c600072d0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c717fa640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c60009eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:38.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c73fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c6000a3f0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:38.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.224+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:38.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.425+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:38.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.425+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:38.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.425+0000 7f6c73fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c6000a3f0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:38.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.425+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c60009eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:38.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.425+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:38.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.826+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.826+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.826+0000 7f6c73fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c6000a3f0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.826+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c60009eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:38.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:38.826+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:39.520 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:00:07.298 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:39.532 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:00:07.344 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:39.534 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:00:07.346 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:39.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:39.627+0000 7f6c72ffd640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.47:3301/0 2024-03-30T07:15:39.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:39.627+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:39.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:39.627+0000 7f6c73fff640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c6000a3f0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-30T07:15:39.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:39.627+0000 7f6c737fe640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c60009eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.52:3300/0 2024-03-30T07:15:39.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:39.627+0000 7f6c737fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:39.845 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:00:07.623 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:39.858 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:00:07.670 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:39.859 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:00:07.671 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:15:41.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): tick 2024-03-30T07:15:41.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): _check_auth_tickets 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 monclient(hunting): continuing hunt 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 -- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 msgr2=0x7f6c60009790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c600072d0 0x7f6c60009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 -- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 msgr2=0x7f6c60009eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c740b4280 0x7f6c60009eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f6c6000a3f0 legacy=0x7f6c60005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient: _add_conns ranks=[3,0] 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-a con 0x7f6c6000d0c0 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c6000f9f0 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740a9210 con 0x7f6c6000f9f0 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:41.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c6000d0c0 2024-03-30T07:15:41.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.224+0000 7f6c717fa640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:41.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.349+0000 7f6c737fe640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:41.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34620/0 (socket says 172.21.15.47:34620) 2024-03-30T07:15:41.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 learned_addr learned my addr 172.21.15.47:0/424875166 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:15:41.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c6000d0c0 auth_method 0 2024-03-30T07:15:41.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:41.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:41.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.383+0000 7f6c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:41.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.407+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:41.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.407+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:41.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.407+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:41.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.407+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.655+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c6000f9f0 legacy=0x7f6c60011de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-30T07:15:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.655+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c6000f9f0 legacy=0x7f6c60011de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T07:15:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.655+0000 7f6c72ffd640 1 --1- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c6000f9f0 0x7f6c60011de0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-30T07:15:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.655+0000 7f6c72ffd640 1 --1- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c6000f9f0 0x7f6c60011de0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-30T07:15:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.655+0000 7f6c727fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-03-30T07:15:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.666+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:41.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c6000d0c0 auth_method 0 2024-03-30T07:15:41.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:41.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:41.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:41.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:41.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:41.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:41.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:41.704+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:42.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.341+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:42.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c6000d0c0 auth_method 0 2024-03-30T07:15:42.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:42.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:42.342+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T07:15:43.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.728+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:43.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.729+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c6000d0c0 auth_method 0 2024-03-30T07:15:43.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.729+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:43.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.729+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:43.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.729+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:43.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.730+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:43.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.730+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:43.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.730+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:43.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:43.730+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): tick 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): _check_auth_tickets 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 monclient(hunting): continuing hunt 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c6000d0c0 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 -- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c6000f9f0 legacy=0x7f6c60011de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c73fff640 1 -- 172.21.15.47:0/424875166 reap_dead start 2024-03-30T07:15:44.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-a con 0x7f6c600072d0 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6c60006230 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c600067a0 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 -- 172.21.15.47:0/424875166 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740aa3c0 con 0x7f6c600067a0 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 -- 172.21.15.47:0/424875166 --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c600072d0 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 1 -- 172.21.15.47:0/424875166 --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740aa630 con 0x7f6c60006230 2024-03-30T07:15:44.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.225+0000 7f6c717fa640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:44.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.233+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:44.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.234+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c600072d0 auth_method 0 2024-03-30T07:15:44.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.234+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:44.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.234+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:44.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.234+0000 7f6c737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:44.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.240+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 msgr2=0x7f6c60009e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:44.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.240+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 msgr2=0x7f6c60009e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:44.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.240+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:44.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.240+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:44.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.247+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:44.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.248+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request con 0x7f6c60006230 auth_method 0 2024-03-30T07:15:44.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.248+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:44.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.248+0000 7f6c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:44.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.248+0000 7f6c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:44.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.249+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-30T07:15:44.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.249+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:44.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.249+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:44.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.249+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:44.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.593+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.593+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request con 0x7f6c60006230 auth_method 0 2024-03-30T07:15:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.593+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.593+0000 7f6c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:44.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.593+0000 7f6c72ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:44.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.595+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-30T07:15:44.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.595+0000 7f6c72ffd640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:44.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.595+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:44.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.595+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:44.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.626+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:44.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.627+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c600072d0 auth_method 0 2024-03-30T07:15:44.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.627+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:44.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.627+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:44.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.627+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:44.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.630+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 msgr2=0x7f6c60009e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:44.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.630+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 msgr2=0x7f6c60009e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:44.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.630+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:44.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.630+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T07:15:44.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572844874 0 0) 0x7f6c740aa3c0 con 0x7f6c600067a0 2024-03-30T07:15:44.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 10 monclient(hunting): _negotiate 2024-03-30T07:15:44.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:44.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:44.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 10 monclient(hunting): my global_id is 4101 2024-03-30T07:15:44.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.907+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c5c003520 con 0x7f6c600067a0 2024-03-30T07:15:44.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.976+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1849147158 0 0) 0x7f6c5c003520 con 0x7f6c600067a0 2024-03-30T07:15:44.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 msgr2=0x7f6c6000f4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:44.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.52:3300/0 conn(0x7f6c60006230 0x7f6c6000f4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:44.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 msgr2=0x7f6c60009e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 --2- 172.21.15.47:0/424875166 >> v2:172.21.15.47:3300/0 conn(0x7f6c600072d0 0x7f6c60009e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _finish_hunting 0 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 monclient: found mon.noname-a-legacy 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.47:6789/0 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60009850 con 0x7f6c600067a0 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _finish_auth 0 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _check_auth_tickets 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f6c6c003fd0 con 0x7f6c600067a0 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-30T07:15:44.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: dump: 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] but i'm connected to v1:172.21.15.47:6789/0, reconnecting 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:44.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 >> v1:172.21.15.47:6789/0 conn(0x7f6c600067a0 legacy=0x7f6c60003ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 0x7f6c5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): picked mon.a con 0x7f6c5c003970 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c5c0047c0 0x7f6c5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): picked mon.c con 0x7f6c5c0047c0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c005180 0x7f6c5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): picked mon.b con 0x7f6c5c005180 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740b4660 con 0x7f6c5c003970 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740aa630 con 0x7f6c5c005180 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c00c0d0 con 0x7f6c5c0047c0 2024-03-30T07:15:44.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:44.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:44.977+0000 7f6c727fc640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:15:45.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.004+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 0x7f6c5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:45.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.007+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c5c003970 auth_method 0 2024-03-30T07:15:45.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.007+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:45.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.007+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:45.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.007+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:45.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.011+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:45.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.011+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:45.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.011+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 10 monclient(hunting): handle_auth_done global_id 4101 payload 293 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c5c0047c0 msgr2=0x7f6c5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c5c0047c0 0x7f6c5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c005180 msgr2=0x7f6c5c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c005180 0x7f6c5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 10 monclient: _finish_hunting 0 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 monclient: found mon.a 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c73fff640 1 -- 172.21.15.47:0/424875166 reap_dead start 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 1 -- 172.21.15.47:0/424875166 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c003520 con 0x7f6c5c003970 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 10 monclient: _finish_auth 0 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 10 monclient: _check_auth_tickets 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c79ac4640 5 monclient: authenticate success, global_id 4101 2024-03-30T07:15:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.024+0000 7f6c79ac4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:15:45.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.048+0000 7f6c737fe640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 0x7f6c5c003d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c64003bf0 tx=0x7f6c5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bcb3598e5401e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6402dd60 con 0x7f6c5c003970 2024-03-30T07:15:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:45.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: dump: 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64031070 con 0x7f6c5c003970 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 1 -- 172.21.15.47:0/424875166 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64037640 con 0x7f6c5c003970 2024-03-30T07:15:45.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 4 set_mon_vals no callback set 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c727fc640 10 monclient: dump: 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:45.066 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 10 monclient: shutdown 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 -- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 msgr2=0x7f6c5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 0x7f6c5c003d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c64003bf0 tx=0x7f6c5c0057f0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 -- 172.21.15.47:0/424875166 shutdown_connections 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/424875166 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c5c003970 0x7f6c5c003d90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 -- 172.21.15.47:0/424875166 >> 172.21.15.47:0/424875166 conn(0x7f6c74065e00 msgr2=0x7f6c74066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 -- 172.21.15.47:0/424875166 shutdown_connections 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.064+0000 7f6c79ac4640 1 -- 172.21.15.47:0/424875166 wait complete. 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.065+0000 7f6c79ac4640 10 monclient: build_initial_monmap 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.065+0000 7f6c79ac4640 10 monclient: monmap: 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:45.069 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 librados: starting msgr at 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 librados: starting objecter 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 Processor -- start 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 -- start start 2024-03-30T07:15:45.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 librados: setting wanted keys 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 1 librados: calling monclient init 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.066+0000 7f6c79ac4640 10 monclient: init 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c74136150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c5c003970 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c740b0820 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c741d9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c741d95f0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c74136bd0 con 0x7f6c740b0820 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c741d9fb0 con 0x7f6c5c003970 2024-03-30T07:15:45.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c741da120 con 0x7f6c741d95f0 2024-03-30T07:15:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:45.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.067+0000 7f6c79ac4640 10 monclient(hunting): authenticate will time out at 1471.182877s 2024-03-30T07:15:45.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.083+0000 7f6c73fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c741d9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.083+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34714/0 (socket says 172.21.15.47:34714) 2024-03-30T07:15:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 learned_addr learned my addr 172.21.15.47:0/2452522987 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:15:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request con 0x7f6c740b0820 auth_method 0 2024-03-30T07:15:45.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.084+0000 7f6c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.085+0000 7f6c72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.085+0000 7f6c72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:45.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.085+0000 7f6c72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 msgr2=0x7f6c741d9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c741d9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 msgr2=0x7f6c74136150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c74136150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 10 monclient: _finish_hunting 0 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 monclient: found mon.noname-a 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:45.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.089+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74143bd0 con 0x7f6c740b0820 2024-03-30T07:15:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.093+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6c680056f0 tx=0x7f6c68009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.093+0000 7f6c5bfff640 10 client.?.objecter ms_handle_connect 0x7f6c740b0820 2024-03-30T07:15:45.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.093+0000 7f6c5bfff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:45.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c68019070 con 0x7f6c740b0820 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: dump: 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:45.136 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: _finish_auth 0 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: _check_auth_tickets 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c68011040 con 0x7f6c740b0820 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 5 monclient: authenticate success, global_id 4107 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 10 monclient: _renew_subs 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c74143dc0 con 0x7f6c740b0820 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c680153f0 con 0x7f6c740b0820 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c5bfff640 10 monclient: dump: 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:45.137 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c71ffb640 4 set_mon_vals no callback set 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 10 monclient: _renew_subs 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c741442d0 con 0x7f6c740b0820 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.134+0000 7f6c79ac4640 1 librados: init done 2024-03-30T07:15:45.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.135+0000 7f6c79ac4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.135+0000 7f6c79ac4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.135+0000 7f6c79ac4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:45.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.135+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c741da260 con 0x7f6c740b0820 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 msgr2=0x7f6c74136690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f6c68015850 con 0x7f6c740b0820 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c72ffd640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 msgr2=0x7f6c74136690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6c680056f0 tx=0x7f6c68009f20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c72ffd640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6c680056f0 tx=0x7f6c68009f20 comp rx=0 tx=0).stop 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 msgr2=0x7f6c74136690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c740b0820 0x7f6c74136690 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c6000c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): picked mon.b con 0x7f6c5c003970 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c60002c40 0x7f6c60008e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): picked mon.a con 0x7f6c60002c40 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c6000e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): picked mon.c con 0x7f6c741d95f0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6000ed90 con 0x7f6c60002c40 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6000ef00 con 0x7f6c5c003970 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6000f070 con 0x7f6c741d95f0 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.350+0000 7f6c5bfff640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c73fff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c6000e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:45.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c737fe640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c6000c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c737fe640 10 monclient(hunting): get_auth_request con 0x7f6c5c003970 auth_method 0 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c737fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c73fff640 10 monclient(hunting): get_auth_request con 0x7f6c741d95f0 auth_method 0 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c73fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:45.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.351+0000 7f6c73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c73fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:15:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:45.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.352+0000 7f6c737fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 293 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 msgr2=0x7f6c6000c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c6000c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c60002c40 msgr2=0x7f6c60008e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c60002c40 0x7f6c60008e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _finish_hunting 0 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 monclient: found mon.c 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f6c6000f230 con 0x7f6c741d95f0 2024-03-30T07:15:45.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c60009850 con 0x7f6c741d95f0 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _finish_auth 0 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 10 monclient: _check_auth_tickets 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.354+0000 7f6c73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:45.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.356+0000 7f6c73fff640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c6000e620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00daf0 tx=0x7f6c68003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:45.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.356+0000 7f6c5bfff640 10 client.4107.objecter ms_handle_connect 0x7f6c741d95f0 2024-03-30T07:15:45.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.356+0000 7f6c5bfff640 10 client.4107.objecter resend_mon_ops 2024-03-30T07:15:45.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c00de50 con 0x7f6c741d95f0 2024-03-30T07:15:45.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:45.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:45.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: dump: 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c01b070 con 0x7f6c741d95f0 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f6c6c014070 con 0x7f6c741d95f0 2024-03-30T07:15:45.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 4 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6c6c00e800 con 0x7f6c741d95f0 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 client.4107.objecter ms_dispatch 0x7f6c74000b90 osd_map(1..1 src has 1..1) v4 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c717fa640 4 set_mon_vals no callback set 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f6c6c00d4d0 con 0x7f6c741d95f0 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:45.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.545+0000 7f6c5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:45.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.611+0000 7f6c79ac4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-30T07:15:45.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.611+0000 7f6c79ac4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-30T07:15:45.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.611+0000 7f6c79ac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:45.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.611+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6c7404e030 con 0x7f6c741d95f0 2024-03-30T07:15:45.628 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:45.627+0000 7fec5ce14040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T07:15:45.640 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:45.639+0000 7efc18c36040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T07:15:45.683 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:45.682+0000 7fec5ce14040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T07:15:45.697 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:45.697+0000 7efc18c36040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T07:15:45.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c5bfff640 1 -- 172.21.15.47:0/2452522987 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6c6c00eb30 con 0x7f6c741d95f0 2024-03-30T07:15:45.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-30T07:15:45.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c5bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-30T07:15:45.833 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 10 librados: watch_flush enter 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 10 librados: watch_flush exit 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 10 client.4107.objecter successfully canceled tick 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 10 monclient: shutdown 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 msgr2=0x7f6c6000e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c6000e620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00daf0 tx=0x7f6c68003e30 comp rx=0 tx=0).stop 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 shutdown_connections 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c741d95f0 0x7f6c6000e620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c60002c40 0x7f6c60008e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 --2- 172.21.15.47:0/2452522987 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c5c003970 0x7f6c6000c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:45.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.832+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 >> 172.21.15.47:0/2452522987 conn(0x7f6c74065e00 msgr2=0x7f6c74139790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:45.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.833+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 shutdown_connections 2024-03-30T07:15:45.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.833+0000 7f6c79ac4640 1 -- 172.21.15.47:0/2452522987 wait complete. 2024-03-30T07:15:45.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:45.833+0000 7f6c79ac4640 1 librados: shutdown 2024-03-30T07:15:45.844 INFO:tasks.ceph:check_enable_crimson: False 2024-03-30T07:15:45.845 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-30T07:15:45.845 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:45.845 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-30T07:15:45.873 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:45.874 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-30T07:15:45.940 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:45.940 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-30T07:15:46.008 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:15:46.008 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-30T07:15:46.075 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:46.075 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-30T07:15:46.102 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:46.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-30T07:15:46.169 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:46.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-30T07:15:46.233 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.232+0000 7fec5ce14040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T07:15:46.235 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-30T07:15:46.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-30T07:15:46.289 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.289+0000 7fec5ce14040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T07:15:46.296 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.296+0000 7efc18c36040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T07:15:46.302 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new bf658e45-24f6-490b-825c-7331119bd2d6 0 2024-03-30T07:15:46.337 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.336+0000 7fec5ce14040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T07:15:46.350 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.350+0000 7efc18c36040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T07:15:46.393 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.392+0000 7fec5ce14040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T07:15:46.396 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.396+0000 7efc18c36040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T07:15:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.435+0000 7f8718700640 10 monclient: get_monmap_and_config 2024-03-30T07:15:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.435+0000 7f8718700640 10 monclient: build_initial_monmap 2024-03-30T07:15:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.435+0000 7f8718700640 10 monclient: monmap: 2024-03-30T07:15:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:46.437133+0000 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:46.437133+0000 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.436+0000 7f8718700640 1 Processor -- start 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.436+0000 7f8718700640 1 -- start start 2024-03-30T07:15:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.436+0000 7f8718700640 10 monclient: init 2024-03-30T07:15:46.439 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.437+0000 7fec5ce14040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): picked mon.noname-a con 0x7f87100b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8710144ab0 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f87100b0d20 0x7f871014be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): picked mon.noname-c con 0x7f87100b0d20 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100a8bd0 con 0x7f8710144ab0 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87100aa4f0 con 0x7f87100b0120 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:46.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8710144f10 con 0x7f87100b0d20 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8718700640 10 monclient(hunting): authenticate will time out at 1482.583363s 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.437+0000 7f8716c9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f87100b0d20 0x7f871014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f871649c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f87100b0d20 0x7f871014be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33910/0 (socket says 172.21.15.52:33910) 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 1 -- 172.21.15.52:0/3475259206 learned_addr learned my addr 172.21.15.52:0/3475259206 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): get_auth_request con 0x7f87100b0d20 auth_method 0 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:46.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.438+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:46.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.439+0000 7f871649c640 1 -- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 msgr2=0x7f87100a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-30T07:15:46.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.439+0000 7f871649c640 1 -- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 msgr2=0x7f87100a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-30T07:15:46.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.439+0000 7f871649c640 1 --2- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:46.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.439+0000 7f871649c640 1 --2- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:46.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 10 monclient(hunting): handle_auth_done global_id 4102 payload 274 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 1 -- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 msgr2=0x7f87100a84a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 1 --2- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 1 -- 172.21.15.52:0/3475259206 >> v1:172.21.15.47:6789/0 conn(0x7f8710144ab0 legacy=0x7f87100712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 10 monclient: _finish_hunting 0 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 1 monclient: found mon.noname-c 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.440+0000 7f8716c9d640 1 -- 172.21.15.52:0/3475259206 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87100b3780 con 0x7f87100b0d20 2024-03-30T07:15:46.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.442+0000 7f8716c9d640 1 --2- 172.21.15.52:0/3475259206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f871014be20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87100b2b00 tx=0x7f870802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3597b1fc1defab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 1 -- 172.21.15.52:0/3475259206 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870802ca80 con 0x7f87100b0d20 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: dump: 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:46.448 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: _finish_auth 0 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: _check_auth_tickets 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 1 -- 172.21.15.52:0/3475259206 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870802cbe0 con 0x7f87100b0d20 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f8718700640 5 monclient: authenticate success, global_id 4102 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f8718700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:15:46.449 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 1 -- 172.21.15.52:0/3475259206 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87080343f0 con 0x7f87100b0d20 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f871549a640 10 monclient: dump: 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.450 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f8718700640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:46.451 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.447+0000 7f8718700640 4 set_mon_vals no callback set 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 10 monclient: shutdown 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 -- 172.21.15.52:0/3475259206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 msgr2=0x7f871014be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 --2- 172.21.15.52:0/3475259206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f871014be20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87100b2b00 tx=0x7f870802bee0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 -- 172.21.15.52:0/3475259206 shutdown_connections 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 --2- 172.21.15.52:0/3475259206 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f871014be20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.454 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 --2- 172.21.15.52:0/3475259206 >> v2:172.21.15.47:3300/0 conn(0x7f87100b0120 0x7f87100a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 -- 172.21.15.52:0/3475259206 >> 172.21.15.52:0/3475259206 conn(0x7f87100b70b0 msgr2=0x7f87100b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 -- 172.21.15.52:0/3475259206 shutdown_connections 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 1 -- 172.21.15.52:0/3475259206 wait complete. 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 10 monclient: build_initial_monmap 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.448+0000 7f8718700640 10 monclient: monmap: 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:46.455 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 librados: starting msgr at 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 librados: starting objecter 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 Processor -- start 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 -- start start 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 librados: setting wanted keys 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 1 librados: calling monclient init 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.449+0000 7f8718700640 10 monclient: init 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 0x7f87100ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): picked mon.noname-c con 0x7f87100b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:46.456 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f87101d9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): picked mon.noname-b con 0x7f87100b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8710144ab0 0x7f87101d9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): picked mon.noname-a con 0x7f8710144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100abe40 con 0x7f8710144ab0 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87100acf40 con 0x7f87100b0d20 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87101420e0 con 0x7f87100b0120 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8718700640 10 monclient(hunting): authenticate will time out at 1482.596056s 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f87101d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f871649c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 0x7f87100ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:46.457 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f87101d9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:33916/0 (socket says 172.21.15.52:33916) 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 1 -- 172.21.15.52:0/82600948 learned_addr learned my addr 172.21.15.52:0/82600948 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 10 monclient(hunting): get_auth_request con 0x7f87100b0d20 auth_method 0 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8715c9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.450+0000 7f8716c9d640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8710144ab0 0x7f87101d9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): get_auth_request con 0x7f87100b0120 auth_method 0 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): get_auth_request con 0x7f8710144ab0 auth_method 0 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:46.458 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8715c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8715c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8715c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f871649c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.451+0000 7f8716c9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:46.460 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.454+0000 7efc18c36040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T07:15:46.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-03-30T07:15:46.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 -- 172.21.15.52:0/82600948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 msgr2=0x7f87101d9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:46.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f87101d9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 -- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8710144ab0 msgr2=0x7f87101d9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8710144ab0 0x7f87101d9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 10 monclient: _finish_hunting 0 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 monclient: found mon.noname-c 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.454+0000 7f871649c640 1 -- 172.21.15.52:0/82600948 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8708009b30 con 0x7f87100b0120 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.457+0000 7f871649c640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 0x7f87100ab7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c7f0 tx=0x7f870c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.457+0000 7f8706ffd640 10 client.?.objecter ms_handle_connect 0x7f87100b0120 2024-03-30T07:15:46.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.457+0000 7f8706ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c007820 con 0x7f87100b0120 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: dump: 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:46.463 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: _finish_auth 0 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870c0079a0 con 0x7f87100b0120 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 5 monclient: authenticate success, global_id 4103 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 10 monclient: _renew_subs 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87100ac120 con 0x7f87100b0120 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c007d50 con 0x7f87100b0120 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8706ffd640 10 monclient: dump: 2024-03-30T07:15:46.464 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 10 monclient: _renew_subs 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8714c99640 4 set_mon_vals no callback set 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87100aaae0 con 0x7f87100b0120 2024-03-30T07:15:46.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.460+0000 7f8718700640 1 librados: init done 2024-03-30T07:15:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.462+0000 7f8718700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.462+0000 7f8718700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.462+0000 7f8718700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.462+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871004e030 con 0x7f87100b0120 2024-03-30T07:15:46.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f870c004020 con 0x7f87100b0120 2024-03-30T07:15:46.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f870c003610 con 0x7f87100b0120 2024-03-30T07:15:46.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 10 client.4103.objecter ms_dispatch 0x7f8710000b90 osd_map(2..2 src has 1..2) v4 2024-03-30T07:15:46.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 3 client.4103.objecter handle_osd_map got epochs [2,2] > 0 2024-03-30T07:15:46.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 3 client.4103.objecter handle_osd_map decoding full epoch 2 2024-03-30T07:15:46.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.464+0000 7f8706ffd640 20 client.4103.objecter dump_active .. 0 homeless 2024-03-30T07:15:46.500 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.500+0000 7efc18c36040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T07:15:46.560 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.559+0000 7fec5ce14040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T07:15:46.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.569+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f870c01f060 con 0x7f87100b0120 2024-03-30T07:15:46.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.569+0000 7f8706ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:46.570 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.569+0000 7f8706ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:46.620 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.620+0000 7efc18c36040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T07:15:46.677 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.676+0000 7fec5ce14040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T07:15:46.693 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.692+0000 7efc18c36040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T07:15:46.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.723+0000 7f8718700640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bf658e45-24f6-490b-825c-7331119bd2d6", "id": 0}] 2024-03-30T07:15:46.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.723+0000 7f8718700640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bf658e45-24f6-490b-825c-7331119bd2d6", "id": 0}] 2024-03-30T07:15:46.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.723+0000 7f8718700640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:46.724 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.723+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bf658e45-24f6-490b-825c-7331119bd2d6", "id": 0} v 0) v1 -- 0x7f87100aadc0 con 0x7f87100b0120 2024-03-30T07:15:46.741 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.740+0000 7fec5ce14040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T07:15:46.743 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.742+0000 7efc18c36040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T07:15:46.794 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.793+0000 7fec5ce14040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T07:15:46.835 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.834+0000 7efc18c36040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T07:15:46.843 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:46.842+0000 7fec5ce14040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T07:15:46.891 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:46.891+0000 7efc18c36040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T07:15:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8706ffd640 1 -- 172.21.15.52:0/82600948 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf658e45-24f6-490b-825c-7331119bd2d6", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f870c0270b0 con 0x7f87100b0120 2024-03-30T07:15:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8706ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bf658e45-24f6-490b-825c-7331119bd2d6", "id": 0}] 2024-03-30T07:15:46.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8706ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:46.925 INFO:teuthology.orchestra.run.smithi052.stdout:0 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8718700640 10 librados: watch_flush enter 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8718700640 10 librados: watch_flush exit 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8718700640 20 client.4103.objecter shutdown clearing up homeless session... 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.924+0000 7f8718700640 10 client.4103.objecter successfully canceled tick 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 10 monclient: shutdown 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 msgr2=0x7f87100ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 0x7f87100ab7c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c7f0 tx=0x7f870c00ccb0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 shutdown_connections 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8710144ab0 0x7f87101d9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.927 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f87100b0d20 0x7f87101d9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 --2- 172.21.15.52:0/82600948 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87100b0120 0x7f87100ab7c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:46.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 >> 172.21.15.52:0/82600948 conn(0x7f87100b70b0 msgr2=0x7f87100b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:46.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 shutdown_connections 2024-03-30T07:15:46.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 -- 172.21.15.52:0/82600948 wait complete. 2024-03-30T07:15:46.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:46.925+0000 7f8718700640 1 librados: shutdown 2024-03-30T07:15:46.936 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 1a3ae202-ca24-4165-affc-cf9ee8066e8c 1 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 10 monclient: get_monmap_and_config 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 10 monclient: monmap: 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.039 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:47.035452+0000 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:47.035452+0000 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:47.040 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 1 Processor -- start 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 1 -- start start 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.034+0000 7f862c773640 10 monclient: init 2024-03-30T07:15:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 0x7f86240a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): picked mon.noname-a con 0x7f86240b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8624144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): picked mon.noname-b con 0x7f86240b0d20 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:47.042 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86240a8bd0 con 0x7f8624144ab0 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86240aa4f0 con 0x7f86240b0120 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8624144f10 con 0x7f86240b0d20 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.035+0000 7f862c773640 10 monclient(hunting): authenticate will time out at 1483.181644s 2024-03-30T07:15:47.043 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862a50f640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 0x7f86240a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.52:40666/0 (socket says 172.21.15.52:40666) 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 1 -- 172.21.15.52:0/3172149655 learned_addr learned my addr 172.21.15.52:0/3172149655 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 10 monclient(hunting): get_auth_request con 0x7f86240b0d20 auth_method 0 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.044 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862ad10640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862a50f640 10 monclient(hunting): get_auth_request con 0x7f86240b0120 auth_method 0 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862a50f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862a50f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.036+0000 7f862a50f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.045 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.037+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-03-30T07:15:47.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 1 -- 172.21.15.52:0/3172149655 >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 msgr2=0x7f86240a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:47.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 1 --2- 172.21.15.52:0/3172149655 >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 0x7f86240a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 1 -- 172.21.15.52:0/3172149655 >> v1:172.21.15.52:6789/0 conn(0x7f8624144ab0 legacy=0x7f86240712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 1 monclient: found mon.noname-b 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862ad10640 1 -- 172.21.15.52:0/3172149655 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86240b3780 con 0x7f86240b0d20 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.039+0000 7f862a50f640 1 --2- 172.21.15.52:0/3172149655 >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 0x7f86240a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:15:47.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.042+0000 7f862ad10640 1 --2- 172.21.15.52:0/3172149655 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f861c009e50 tx=0x7f861c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db90a609fcca8e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 1 -- 172.21.15.52:0/3172149655 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c02ca70 con 0x7f86240b0d20 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: dump: 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.056 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.057 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 1 -- 172.21.15.52:0/3172149655 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c02cbd0 con 0x7f86240b0d20 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 1 -- 172.21.15.52:0/3172149655 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c0343f0 con 0x7f86240b0d20 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862c773640 5 monclient: authenticate success, global_id 4106 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862c773640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862c773640 4 set_mon_vals no callback set 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.046+0000 7f862950d640 10 monclient: dump: 2024-03-30T07:15:47.059 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.060 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 10 monclient: shutdown 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 -- 172.21.15.52:0/3172149655 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 msgr2=0x7f862414be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 --2- 172.21.15.52:0/3172149655 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f861c009e50 tx=0x7f861c02bed0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 -- 172.21.15.52:0/3172149655 shutdown_connections 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 --2- 172.21.15.52:0/3172149655 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0d20 0x7f862414be20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 --2- 172.21.15.52:0/3172149655 >> v2:172.21.15.47:3300/0 conn(0x7f86240b0120 0x7f86240a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 -- 172.21.15.52:0/3172149655 >> 172.21.15.52:0/3172149655 conn(0x7f86240b70b0 msgr2=0x7f86240b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 -- 172.21.15.52:0/3172149655 shutdown_connections 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.047+0000 7f862c773640 1 -- 172.21.15.52:0/3172149655 wait complete. 2024-03-30T07:15:47.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 10 monclient: monmap: 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.063 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 librados: starting msgr at 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 librados: starting objecter 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 Processor -- start 2024-03-30T07:15:47.064 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 -- start start 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 librados: setting wanted keys 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 1 librados: calling monclient init 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.048+0000 7f862c773640 10 monclient: init 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 0x7f86240ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.065 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): picked mon.noname-c con 0x7f86240b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86240b0d20 0x7f86240abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): picked mon.noname-b con 0x7f86240b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 0x7f86240aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): picked mon.noname-a con 0x7f8624144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240ac2e0 con 0x7f8624144ab0 2024-03-30T07:15:47.066 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86241d4050 con 0x7f86240b0d20 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240aa830 con 0x7f86240b0120 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.049+0000 7f862c773640 10 monclient(hunting): authenticate will time out at 1483.195808s 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 0x7f86240ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.067 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862ad10640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 0x7f86240aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 0x7f86240ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.52:40678/0 (socket says 172.21.15.52:40678) 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 1 -- 172.21.15.52:0/1642990900 learned_addr learned my addr 172.21.15.52:0/1642990900 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 10 monclient(hunting): get_auth_request con 0x7f86240b0120 auth_method 0 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862a50f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.068 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862ad10640 10 monclient(hunting): get_auth_request con 0x7f8624144ab0 auth_method 0 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862ad10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862ad10640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.050+0000 7f862ad10640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862a50f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.069 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.051+0000 7f862ad10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86240b0d20 0x7f86240abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): get_auth_request con 0x7f86240b0d20 auth_method 0 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.052+0000 7f8629d0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.053+0000 7f862ad10640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 -- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 msgr2=0x7f86240ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 0x7f86240ab000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 -- 172.21.15.52:0/1642990900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86240b0d20 msgr2=0x7f86240abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86240b0d20 0x7f86240abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 monclient: found mon.noname-a 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.054+0000 7f862ad10640 1 -- 172.21.15.52:0/1642990900 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8620009400 con 0x7f8624144ab0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.056+0000 7f862ad10640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 0x7f86240aedd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f861c032000 tx=0x7f861c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.056+0000 7f8616ffd640 10 client.?.objecter ms_handle_connect 0x7f8624144ab0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.056+0000 7f8616ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c009510 con 0x7f8624144ab0 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: dump: 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.073 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c03e450 con 0x7f8624144ab0 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c03e800 con 0x7f8624144ab0 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.074 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8616ffd640 10 monclient: dump: 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 5 monclient: authenticate success, global_id 4119 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 10 monclient: _renew_subs 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.075 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c009b30 con 0x7f8624144ab0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f8628d0c640 4 set_mon_vals no callback set 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 10 monclient: _renew_subs 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86240af4f0 con 0x7f8624144ab0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.061+0000 7f862c773640 1 librados: init done 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.062+0000 7f862c773640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.062+0000 7f862c773640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.062+0000 7f862c773640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.062+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86240ac420 con 0x7f8624144ab0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f861c03ec60 con 0x7f8624144ab0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f861c051ce0 con 0x7f8624144ab0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 10 client.4119.objecter ms_dispatch 0x7f8624000b90 osd_map(3..3 src has 1..3) v4 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 3 client.4119.objecter handle_osd_map got epochs [3,3] > 0 2024-03-30T07:15:47.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 3 client.4119.objecter handle_osd_map decoding full epoch 3 2024-03-30T07:15:47.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.066+0000 7f8616ffd640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-30T07:15:47.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.185+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f861c043020 con 0x7f8624144ab0 2024-03-30T07:15:47.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.185+0000 7f8616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.186 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.185+0000 7f8616ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:47.244 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.243+0000 7fec5ce14040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T07:15:47.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.258+0000 7f862c773640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1a3ae202-ca24-4165-affc-cf9ee8066e8c", "id": 1}] 2024-03-30T07:15:47.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.258+0000 7f862c773640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1a3ae202-ca24-4165-affc-cf9ee8066e8c", "id": 1}] 2024-03-30T07:15:47.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.258+0000 7f862c773640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.258+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1a3ae202-ca24-4165-affc-cf9ee8066e8c", "id": 1} v 0) v1 -- 0x7f862404e030 con 0x7f8624144ab0 2024-03-30T07:15:47.326 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.326+0000 7efc18c36040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T07:15:47.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.361+0000 7f8616ffd640 1 -- 172.21.15.52:0/1642990900 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a3ae202-ca24-4165-affc-cf9ee8066e8c", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f861c051430 con 0x7f8624144ab0 2024-03-30T07:15:47.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.361+0000 7f8616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1a3ae202-ca24-4165-affc-cf9ee8066e8c", "id": 1}] 2024-03-30T07:15:47.362 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.361+0000 7f8616ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:47.363 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 10 librados: watch_flush enter 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 10 librados: watch_flush exit 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 10 client.4119.objecter successfully canceled tick 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 10 monclient: shutdown 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 msgr2=0x7f86240aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 0x7f86240aedd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f861c032000 tx=0x7f861c004a40 comp rx=0 tx=0).stop 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 shutdown_connections 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8624144ab0 0x7f86240aedd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.364 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f86240b0d20 0x7f86240abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 --2- 172.21.15.52:0/1642990900 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86240b0120 0x7f86240ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 >> 172.21.15.52:0/1642990900 conn(0x7f86240b70b0 msgr2=0x7f86240b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 shutdown_connections 2024-03-30T07:15:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 -- 172.21.15.52:0/1642990900 wait complete. 2024-03-30T07:15:47.365 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.362+0000 7f862c773640 1 librados: shutdown 2024-03-30T07:15:47.374 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new b93375d8-53af-4822-ac8f-a660429e26f0 2 2024-03-30T07:15:47.386 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.386+0000 7fec5ce14040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T07:15:47.471 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.470+0000 7efc18c36040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T07:15:47.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.472+0000 7f0ed2a77640 10 monclient: get_monmap_and_config 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.472+0000 7f0ed2a77640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.472+0000 7f0ed2a77640 10 monclient: monmap: 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:47.474131+0000 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:47.474131+0000 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:47.474 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:47.475 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.473+0000 7f0ed2a77640 1 Processor -- start 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.473+0000 7f0ed2a77640 1 -- start start 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.473+0000 7f0ed2a77640 10 monclient: init 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 0x7f0ecc0a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-b con 0x7f0ecc0b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ecc144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0ecc0b0d20 0x7f0ecc14be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc0b0d20 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:47.477 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ecc0a8bd0 con 0x7f0ecc144ab0 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ecc0aa4f0 con 0x7f0ecc0b0d20 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc144f10 con 0x7f0ecc0b0120 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.474+0000 7f0ed2a77640 10 monclient(hunting): authenticate will time out at 1483.620392s 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 0x7f0ecc0a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed1014640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f0ecc0b0d20 0x7f0ecc14be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ecbfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f0ecc144ab0 0x7f0ecc0712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37142/0 (socket says 172.21.15.52:37142) 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ecbfff640 1 -- 172.21.15.52:0/1136524657 learned_addr learned my addr 172.21.15.52:0/1136524657 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): get_auth_request con 0x7f0ecc0b0120 auth_method 0 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed1014640 10 monclient(hunting): get_auth_request con 0x7f0ecc0b0d20 auth_method 0 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed1014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed1014640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed1014640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.475+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 1 -- 172.21.15.52:0/1136524657 >> v2:172.21.15.52:3300/0 conn(0x7f0ecc0b0d20 msgr2=0x7f0ecc14be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 1 --2- 172.21.15.52:0/1136524657 >> v2:172.21.15.52:3300/0 conn(0x7f0ecc0b0d20 0x7f0ecc14be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 1 -- 172.21.15.52:0/1136524657 >> v1:172.21.15.52:6789/0 conn(0x7f0ecc144ab0 legacy=0x7f0ecc0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 1 monclient: found mon.noname-b 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.477+0000 7f0ed0813640 1 -- 172.21.15.52:0/1136524657 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc0b3780 con 0x7f0ecc0b0120 2024-03-30T07:15:47.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.479+0000 7f0ed0813640 1 --2- 172.21.15.52:0/1136524657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 0x7f0ecc0a84a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c016d87937808c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 1 -- 172.21.15.52:0/1136524657 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec002e070 con 0x7f0ecc0b0120 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: dump: 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.484 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 1 -- 172.21.15.52:0/1136524657 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00315a0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 1 -- 172.21.15.52:0/1136524657 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0031930 con 0x7f0ecc0b0120 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ecb7fe640 10 monclient: dump: 2024-03-30T07:15:47.485 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ed2a77640 5 monclient: authenticate success, global_id 4112 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ed2a77640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:47.486 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.483+0000 7f0ed2a77640 4 set_mon_vals no callback set 2024-03-30T07:15:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 10 monclient: shutdown 2024-03-30T07:15:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 -- 172.21.15.52:0/1136524657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 msgr2=0x7f0ecc0a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/1136524657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 0x7f0ecc0a84a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b840 comp rx=0 tx=0).stop 2024-03-30T07:15:47.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 -- 172.21.15.52:0/1136524657 shutdown_connections 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/1136524657 >> v2:172.21.15.52:3300/0 conn(0x7f0ecc0b0d20 0x7f0ecc14be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/1136524657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc0b0120 0x7f0ecc0a84a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 -- 172.21.15.52:0/1136524657 >> 172.21.15.52:0/1136524657 conn(0x7f0ecc0b70b0 msgr2=0x7f0ecc0b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 -- 172.21.15.52:0/1136524657 shutdown_connections 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 1 -- 172.21.15.52:0/1136524657 wait complete. 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.484+0000 7f0ed2a77640 10 monclient: monmap: 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.490 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 librados: starting msgr at 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 librados: starting objecter 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 Processor -- start 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 -- start start 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 librados: setting wanted keys 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 librados: calling monclient init 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient: init 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 0x7f0ecc0ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-a con 0x7f0ecc0b0120 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:47.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ecc0b0d20 0x7f0ecc1d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-b con 0x7f0ecc0b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0acb50 con 0x7f0ecc0b0120 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.485+0000 7f0ed2a77640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0ac6c0 con 0x7f0ecc0b0d20 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed2a77640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed2a77640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0aa960 con 0x7f0ecc144ab0 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed2a77640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed2a77640 10 monclient(hunting): authenticate will time out at 1483.631971s 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed0813640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 0x7f0ecc0ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.52:40704/0 (socket says 172.21.15.52:40704) 2024-03-30T07:15:47.492 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 1 -- 172.21.15.52:0/646402884 learned_addr learned my addr 172.21.15.52:0/646402884 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 10 monclient(hunting): get_auth_request con 0x7f0ecc144ab0 auth_method 0 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.486+0000 7f0ed1014640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.487+0000 7f0ed0813640 10 monclient(hunting): get_auth_request con 0x7f0ecc0b0120 auth_method 0 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.487+0000 7f0ed0813640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.487+0000 7f0ed0813640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.487+0000 7f0ed0813640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed0813640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed1014640 1 -- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 msgr2=0x7f0ecc1d70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-30T07:15:47.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed1014640 1 -- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 msgr2=0x7f0ecc1d70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T07:15:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed1014640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:15:47.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.488+0000 7f0ed1014640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 10 monclient(hunting): handle_auth_done global_id 4122 payload 824 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 -- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 msgr2=0x7f0ecc1d70b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 -- 172.21.15.52:0/646402884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ecc0b0d20 msgr2=0x7f0ecc1d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ecc0b0d20 0x7f0ecc1d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 monclient: found mon.noname-a 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.491+0000 7f0ed0813640 1 -- 172.21.15.52:0/646402884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec0009400 con 0x7f0ecc0b0120 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.494+0000 7f0ed0813640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 0x7f0ecc0ab7a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0033000 tx=0x7f0ec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.494+0000 7f0ec97fa640 10 client.?.objecter ms_handle_connect 0x7f0ecc0b0120 2024-03-30T07:15:47.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.494+0000 7f0ec97fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:47.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0032840 con 0x7f0ecc0b0120 2024-03-30T07:15:47.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:47.499 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: dump: 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00329c0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0032d70 con 0x7f0ecc0b0120 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 5 monclient: authenticate success, global_id 4122 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 10 monclient: _renew_subs 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ecc1d75f0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ecaffd640 4 set_mon_vals no callback set 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ec97fa640 10 monclient: dump: 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 10 monclient: _renew_subs 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.497+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ecc0abec0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.498+0000 7f0ed2a77640 1 librados: init done 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.498+0000 7f0ed2a77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.498+0000 7f0ed2a77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.498+0000 7f0ed2a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.498+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ecc04e030 con 0x7f0ecc0b0120 2024-03-30T07:15:47.504 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.500+0000 7fec5ce14040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T07:15:47.548 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.547+0000 7fec5ce14040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T07:15:47.598 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.598+0000 7efc18c36040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T07:15:47.602 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.601+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f0ec004e730 con 0x7f0ecc0b0120 2024-03-30T07:15:47.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f0ec003b810 con 0x7f0ecc0b0120 2024-03-30T07:15:47.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 10 client.4122.objecter ms_dispatch 0x7f0ecc000b90 osd_map(4..4 src has 1..4) v4 2024-03-30T07:15:47.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 3 client.4122.objecter handle_osd_map got epochs [4,4] > 0 2024-03-30T07:15:47.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 3 client.4122.objecter handle_osd_map decoding full epoch 4 2024-03-30T07:15:47.606 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 20 client.4122.objecter dump_active .. 0 homeless 2024-03-30T07:15:47.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f0ecc0abec0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.607 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.605+0000 7f0ec97fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:47.645 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.645+0000 7efc18c36040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T07:15:47.664 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.663+0000 7fec5ce14040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T07:15:47.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.679+0000 7f0ed2a77640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b93375d8-53af-4822-ac8f-a660429e26f0", "id": 2}] 2024-03-30T07:15:47.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.680+0000 7f0ed2a77640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b93375d8-53af-4822-ac8f-a660429e26f0", "id": 2}] 2024-03-30T07:15:47.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.680+0000 7f0ed2a77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.681 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.680+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b93375d8-53af-4822-ac8f-a660429e26f0", "id": 2} v 0) v1 -- 0x7f0ecc143ea0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ec97fa640 1 -- 172.21.15.52:0/646402884 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b93375d8-53af-4822-ac8f-a660429e26f0", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0ec003b4c0 con 0x7f0ecc0b0120 2024-03-30T07:15:47.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ec97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b93375d8-53af-4822-ac8f-a660429e26f0", "id": 2}] 2024-03-30T07:15:47.729 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ec97fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:47.730 INFO:teuthology.orchestra.run.smithi052.stdout:2 2024-03-30T07:15:47.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 10 librados: watch_flush enter 2024-03-30T07:15:47.732 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 10 librados: watch_flush exit 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 20 client.4122.objecter shutdown clearing up homeless session... 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 10 client.4122.objecter successfully canceled tick 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 10 monclient: shutdown 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 msgr2=0x7f0ecc0ab7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.733 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.728+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 0x7f0ecc0ab7a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0033000 tx=0x7f0ec00046c0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 shutdown_connections 2024-03-30T07:15:47.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ecc144ab0 0x7f0ecc1d70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0ecc0b0d20 0x7f0ecc1d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 --2- 172.21.15.52:0/646402884 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ecc0b0120 0x7f0ecc0ab7a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.734 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 >> 172.21.15.52:0/646402884 conn(0x7f0ecc0b70b0 msgr2=0x7f0ecc143320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:47.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 shutdown_connections 2024-03-30T07:15:47.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 -- 172.21.15.52:0/646402884 wait complete. 2024-03-30T07:15:47.735 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.729+0000 7f0ed2a77640 1 librados: shutdown 2024-03-30T07:15:47.740 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 8fc299ae-e44c-42d3-aa0c-6f7cafad40ae 3 2024-03-30T07:15:47.764 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.764+0000 7efc18c36040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T07:15:47.765 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.764+0000 7fec5ce14040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T07:15:47.862 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:47.861+0000 7efc18c36040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T07:15:47.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.863+0000 7f4ece350640 10 monclient: get_monmap_and_config 2024-03-30T07:15:47.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.863+0000 7f4ece350640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.864 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.863+0000 7f4ece350640 10 monclient: monmap: 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:47.864925+0000 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:47.864925+0000 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:47.865 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.864+0000 7f4ece350640 1 Processor -- start 2024-03-30T07:15:47.867 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.864+0000 7f4ece350640 1 -- start start 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.864+0000 7f4ece350640 10 monclient: init 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4ec80b4280 0x7f4ec80a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80b4280 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ec813e930 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4ec813eda0 0x7f4ec8141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec813eda0 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec80aa3c0 con 0x7f4ec813e930 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec80b4660 con 0x7f4ec813eda0 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec80aa630 con 0x7f4ec80b4280 2024-03-30T07:15:47.868 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ece350640 10 monclient(hunting): authenticate will time out at 1484.011117s 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.865+0000 7f4ecc8ed640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4ec813eda0 0x7f4ec8141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f4ec813e930 0x7f4ec8141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37150/0 (socket says 172.21.15.52:37150) 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec77fe640 1 -- 172.21.15.52:0/10006293 learned_addr learned my addr 172.21.15.52:0/10006293 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec7fff640 1 --2- 172.21.15.52:0/10006293 >> v2:172.21.15.52:3300/0 conn(0x7f4ec80b4280 0x7f4ec80a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): get_auth_request con 0x7f4ec813eda0 auth_method 0 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.869 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request con 0x7f4ec80b4280 auth_method 0 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.866+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_done global_id 4125 payload 274 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/10006293 >> v2:172.21.15.52:3300/0 conn(0x7f4ec80b4280 msgr2=0x7f4ec80a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:47.870 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 1 --2- 172.21.15.52:0/10006293 >> v2:172.21.15.52:3300/0 conn(0x7f4ec80b4280 0x7f4ec80a8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/10006293 >> v1:172.21.15.52:6789/0 conn(0x7f4ec813e930 legacy=0x7f4ec8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 1 monclient: found mon.noname-a 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.868+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/10006293 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec80a9210 con 0x7f4ec813eda0 2024-03-30T07:15:47.871 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.870+0000 7f4ecc8ed640 1 --2- 172.21.15.52:0/10006293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8141d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80b6ff0 tx=0x7f4ec002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eca909c6eb6f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.874 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 1 -- 172.21.15.52:0/10006293 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec002cbf0 con 0x7f4ec813eda0 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: dump: 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.875 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 1 -- 172.21.15.52:0/10006293 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec002cd50 con 0x7f4ec813eda0 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 1 -- 172.21.15.52:0/10006293 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0034660 con 0x7f4ec813eda0 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ece350640 5 monclient: authenticate success, global_id 4125 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ece350640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ece350640 4 set_mon_vals no callback set 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.873+0000 7f4ec6ffd640 10 monclient: dump: 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.876 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.877 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 10 monclient: shutdown 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 -- 172.21.15.52:0/10006293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 msgr2=0x7f4ec8141d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 --2- 172.21.15.52:0/10006293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8141d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80b6ff0 tx=0x7f4ec002be40 comp rx=0 tx=0).stop 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 -- 172.21.15.52:0/10006293 shutdown_connections 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 --2- 172.21.15.52:0/10006293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8141d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.880 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 --2- 172.21.15.52:0/10006293 >> v2:172.21.15.52:3300/0 conn(0x7f4ec80b4280 0x7f4ec80a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 -- 172.21.15.52:0/10006293 >> 172.21.15.52:0/10006293 conn(0x7f4ec8065e00 msgr2=0x7f4ec8066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 -- 172.21.15.52:0/10006293 shutdown_connections 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.874+0000 7f4ece350640 1 -- 172.21.15.52:0/10006293 wait complete. 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 10 monclient: build_initial_monmap 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 10 monclient: monmap: 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:47.881 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 librados: starting msgr at 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 librados: starting objecter 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 Processor -- start 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 -- start start 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 librados: setting wanted keys 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 1 librados: calling monclient init 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.875+0000 7f4ece350640 10 monclient: init 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ec80b4280 0x7f4ec8135460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:47.882 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ec813e930 0x7f4ec81d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec813e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec813eda0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec81d6370 con 0x7f4ec813eda0 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec81d6650 con 0x7f4ec813e930 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8139180 con 0x7f4ec80b4280 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.876+0000 7f4ece350640 10 monclient(hunting): authenticate will time out at 1484.022456s 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ec7fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ec80b4280 0x7f4ec8135460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:47.883 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.52:51294/0 (socket says 172.21.15.52:51294) 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/1482171069 learned_addr learned my addr 172.21.15.52:0/1482171069 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): get_auth_request con 0x7f4ec813eda0 auth_method 0 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request con 0x7f4ec80b4280 auth_method 0 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.877+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.878+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:47.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.878+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:47.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.878+0000 7f4ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:47.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2024-03-30T07:15:47.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ec80b4280 msgr2=0x7f4ec8135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ec80b4280 0x7f4ec8135460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/1482171069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ec813e930 msgr2=0x7f4ec81d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ec813e930 0x7f4ec81d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 10 monclient: _finish_hunting 0 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 monclient: found mon.noname-a 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.880+0000 7f4ecc8ed640 1 -- 172.21.15.52:0/1482171069 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec0009a70 con 0x7f4ec813eda0 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.883+0000 7f4ecc8ed640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009e80 tx=0x7f4ec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.883+0000 7f4ec4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ec813eda0 2024-03-30T07:15:47.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.883+0000 7f4ec4ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0040650 con 0x7f4ec813eda0 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: dump: 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.888 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: _finish_auth 0 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec00407b0 con 0x7f4ec813eda0 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0040b60 con 0x7f4ec813eda0 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec4ff9640 10 monclient: dump: 2024-03-30T07:15:47.889 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 5 monclient: authenticate success, global_id 4128 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 10 monclient: _renew_subs 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec81d70b0 con 0x7f4ec813eda0 2024-03-30T07:15:47.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ec67fc640 4 set_mon_vals no callback set 2024-03-30T07:15:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 10 monclient: _renew_subs 2024-03-30T07:15:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec813a380 con 0x7f4ec813eda0 2024-03-30T07:15:47.891 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.886+0000 7f4ece350640 1 librados: init done 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.887+0000 7f4ece350640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.887+0000 7f4ece350640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.887+0000 7f4ece350640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.887+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec81d7270 con 0x7f4ec813eda0 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f4ec0052430 con 0x7f4ec813eda0 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f4ec003e890 con 0x7f4ec813eda0 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 10 client.4128.objecter ms_dispatch 0x7f4ec8000b90 osd_map(5..5 src has 1..5) v4 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 3 client.4128.objecter handle_osd_map got epochs [5,5] > 0 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 3 client.4128.objecter handle_osd_map decoding full epoch 5 2024-03-30T07:15:47.892 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.889+0000 7f4ec4ff9640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-30T07:15:47.929 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.928+0000 7fec5ce14040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T07:15:47.972 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:47.971+0000 7fec5ce14040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T07:15:47.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.995+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f4ec0044020 con 0x7f4ec813eda0 2024-03-30T07:15:47.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.995+0000 7f4ec4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:47.996 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:47.995+0000 7f4ec4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:48.025 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:48.024+0000 7efc18c36040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T07:15:48.068 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:48.068+0000 7efc18c36040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T07:15:48.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.069+0000 7f4ece350640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8fc299ae-e44c-42d3-aa0c-6f7cafad40ae", "id": 3}] 2024-03-30T07:15:48.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.069+0000 7f4ece350640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8fc299ae-e44c-42d3-aa0c-6f7cafad40ae", "id": 3}] 2024-03-30T07:15:48.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.069+0000 7f4ece350640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.069+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8fc299ae-e44c-42d3-aa0c-6f7cafad40ae", "id": 3} v 0) v1 -- 0x7f4ec804e030 con 0x7f4ec813eda0 2024-03-30T07:15:48.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.122+0000 7f4ec4ff9640 1 -- 172.21.15.52:0/1482171069 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fc299ae-e44c-42d3-aa0c-6f7cafad40ae", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4ec0039070 con 0x7f4ec813eda0 2024-03-30T07:15:48.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.122+0000 7f4ec4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8fc299ae-e44c-42d3-aa0c-6f7cafad40ae", "id": 3}] 2024-03-30T07:15:48.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.122+0000 7f4ec4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:48.123 INFO:teuthology.orchestra.run.smithi052.stdout:3 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 10 librados: watch_flush enter 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 10 librados: watch_flush exit 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 10 client.4128.objecter successfully canceled tick 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 10 monclient: shutdown 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 msgr2=0x7f4ec8139c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0009e80 tx=0x7f4ec0004a40 comp rx=0 tx=0).stop 2024-03-30T07:15:48.125 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 shutdown_connections 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ec813eda0 0x7f4ec8139c60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4ec813e930 0x7f4ec81d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 --2- 172.21.15.52:0/1482171069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ec80b4280 0x7f4ec8135460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 >> 172.21.15.52:0/1482171069 conn(0x7f4ec8065e00 msgr2=0x7f4ec80b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 shutdown_connections 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 -- 172.21.15.52:0/1482171069 wait complete. 2024-03-30T07:15:48.126 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.123+0000 7f4ece350640 1 librados: shutdown 2024-03-30T07:15:48.135 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 4e1deafb-c667-4984-8939-c657168e85da 4 2024-03-30T07:15:48.136 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:48.133+0000 7fec5ce14040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T07:15:48.232 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:48.232+0000 7efc18c36040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T07:15:48.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.234+0000 7fbf23a28640 10 monclient: get_monmap_and_config 2024-03-30T07:15:48.235 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.234+0000 7fbf23a28640 10 monclient: build_initial_monmap 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.234+0000 7fbf23a28640 10 monclient: monmap: 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:48.235426+0000 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:48.235426+0000 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:48.236 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:48.237 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.234+0000 7fbf23a28640 1 Processor -- start 2024-03-30T07:15:48.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 -- start start 2024-03-30T07:15:48.237 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient: init 2024-03-30T07:15:48.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:48.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-03-30T07:15:48.240 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-c con 0x7fbf1c0b0120 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c144ab0 0x7fbf1c0712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-b con 0x7fbf1c144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbf1c0b0d20 0x7fbf1c1415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-a con 0x7fbf1c0b0d20 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c0aa4f0 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1c0a8bd0 con 0x7fbf1c0b0120 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c144f10 con 0x7fbf1c144ab0 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.235+0000 7fbf23a28640 10 monclient(hunting): authenticate will time out at 1484.381608s 2024-03-30T07:15:48.241 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf21fc5640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbf1c0b0d20 0x7fbf1c1415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf20fc3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c144ab0 0x7fbf1c0712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45728/0 (socket says 172.21.15.52:45728) 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 1 -- 172.21.15.52:0/387749685 learned_addr learned my addr 172.21.15.52:0/387749685 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 10 monclient(hunting): get_auth_request con 0x7fbf1c0b0120 auth_method 0 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf217c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf21fc5640 10 monclient(hunting): get_auth_request con 0x7fbf1c0b0d20 auth_method 0 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf21fc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf21fc5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf21fc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.242 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf20fc3640 10 monclient(hunting): get_auth_request con 0x7fbf1c144ab0 auth_method 0 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf20fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf20fc3640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.236+0000 7fbf20fc3640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.237+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_done global_id 4131 payload 274 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 -- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c144ab0 msgr2=0x7fbf1c0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c144ab0 0x7fbf1c0712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.243 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 -- 172.21.15.52:0/387749685 >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 msgr2=0x7fbf1c0a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/387749685 >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 10 monclient: _finish_hunting 0 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 monclient: found mon.noname-a 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf21fc5640 1 -- 172.21.15.52:0/387749685 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0b3780 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.238+0000 7fbf217c4640 1 --2- 172.21.15.52:0/387749685 >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:15:48.244 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.241+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c0b0d20 0x7fbf1c1415b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c071c40 tx=0x7fbf1402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3bd26261f3634b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:48.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 1 -- 172.21.15.52:0/387749685 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1402cbd0 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:48.248 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: dump: 2024-03-30T07:15:48.248 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: _finish_auth 0 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: _check_auth_tickets 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 1 -- 172.21.15.52:0/387749685 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1402cd30 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:48.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 1 -- 172.21.15.52:0/387749685 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1402cea0 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf23a28640 5 monclient: authenticate success, global_id 4131 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf23a28640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf23a28640 4 set_mon_vals no callback set 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.247+0000 7fbf13fff640 10 monclient: dump: 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.250 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.251 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.251 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 10 monclient: shutdown 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 -- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c0b0d20 msgr2=0x7fbf1c1415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 --2- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c0b0d20 0x7fbf1c1415b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c071c40 tx=0x7fbf1402be20 comp rx=0 tx=0).stop 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 -- 172.21.15.52:0/387749685 shutdown_connections 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 --2- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c0b0d20 0x7fbf1c1415b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 --2- 172.21.15.52:0/387749685 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c144ab0 0x7fbf1c0712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 --2- 172.21.15.52:0/387749685 >> v2:172.21.15.52:3300/0 conn(0x7fbf1c0b0120 0x7fbf1c0a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 -- 172.21.15.52:0/387749685 >> 172.21.15.52:0/387749685 conn(0x7fbf1c0b70b0 msgr2=0x7fbf1c0b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 -- 172.21.15.52:0/387749685 shutdown_connections 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.248+0000 7fbf23a28640 1 -- 172.21.15.52:0/387749685 wait complete. 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 10 monclient: build_initial_monmap 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 10 monclient: monmap: 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:48.255 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 librados: starting msgr at 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 librados: starting objecter 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 Processor -- start 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 -- start start 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 librados: setting wanted keys 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 1 librados: calling monclient init 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.249+0000 7fbf23a28640 10 monclient: init 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 0x7fbf1c0ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-b con 0x7fbf1c0b0120 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c0b0d20 0x7fbf1c0ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-c con 0x7fbf1c0b0d20 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:48.256 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 0x7fbf1c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): picked mon.noname-a con 0x7fbf1c144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1d9b00 con 0x7fbf1c144ab0 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1d9de0 con 0x7fbf1c0b0120 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c0afa20 con 0x7fbf1c0b0d20 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf23a28640 10 monclient(hunting): authenticate will time out at 1484.396139s 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf217c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 0x7fbf1c0ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf20fc3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c0b0d20 0x7fbf1c0ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.250+0000 7fbf21fc5640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 0x7fbf1c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 0x7fbf1c0ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45734/0 (socket says 172.21.15.52:45734) 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 1 -- 172.21.15.52:0/900556622 learned_addr learned my addr 172.21.15.52:0/900556622 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:48.257 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): get_auth_request con 0x7fbf1c0b0120 auth_method 0 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): get_auth_request con 0x7fbf1c144ab0 auth_method 0 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): get_auth_request con 0x7fbf1c0b0d20 auth_method 0 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.258 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf217c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.251+0000 7fbf20fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 -- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c0b0d20 msgr2=0x7fbf1c0ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c0b0d20 0x7fbf1c0ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 -- 172.21.15.52:0/900556622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 msgr2=0x7fbf1c0ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 0x7fbf1c0ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 10 monclient: _finish_hunting 0 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 monclient: found mon.noname-a 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.254+0000 7fbf21fc5640 1 -- 172.21.15.52:0/900556622 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf180094d0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.257+0000 7fbf21fc5640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 0x7fbf1c0af020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf14033000 tx=0x7fbf14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:48.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.257+0000 7fbf11ffb640 10 client.?.objecter ms_handle_connect 0x7fbf1c144ab0 2024-03-30T07:15:48.261 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.257+0000 7fbf11ffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:48.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1402cea0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.268 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: dump: 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.269 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: _finish_auth 0 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf14034e40 con 0x7fbf1c144ab0 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1403f640 con 0x7fbf1c144ab0 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf23a28640 5 monclient: authenticate success, global_id 4134 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf23a28640 10 monclient: _renew_subs 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf23a28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf14009a70 con 0x7fbf1c144ab0 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf137fe640 4 set_mon_vals no callback set 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.270 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.266+0000 7fbf11ffb640 10 monclient: dump: 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.271 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.267+0000 7fbf23a28640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.267+0000 7fbf23a28640 10 monclient: _renew_subs 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.267+0000 7fbf23a28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.267+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c1da4e0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.267+0000 7fbf23a28640 1 librados: init done 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.268+0000 7fbf23a28640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.268+0000 7fbf23a28640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.272 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.268+0000 7fbf23a28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.273 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.268+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1c0b15c0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fbf1403faa0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fbf140473c0 con 0x7fbf1c144ab0 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 10 client.4134.objecter ms_dispatch 0x7fbf1c000b90 osd_map(6..6 src has 1..6) v4 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 3 client.4134.objecter handle_osd_map got epochs [6,6] > 0 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 3 client.4134.objecter handle_osd_map decoding full epoch 6 2024-03-30T07:15:48.275 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.274+0000 7fbf11ffb640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-30T07:15:48.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.397+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fbf1404d060 con 0x7fbf1c144ab0 2024-03-30T07:15:48.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.397+0000 7fbf11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.398 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.397+0000 7fbf11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:48.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.471+0000 7fbf23a28640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e1deafb-c667-4984-8939-c657168e85da", "id": 4}] 2024-03-30T07:15:48.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.472+0000 7fbf23a28640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e1deafb-c667-4984-8939-c657168e85da", "id": 4}] 2024-03-30T07:15:48.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.472+0000 7fbf23a28640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:48.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.472+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e1deafb-c667-4984-8939-c657168e85da", "id": 4} v 0) v1 -- 0x7fbf1c1d9c40 con 0x7fbf1c144ab0 2024-03-30T07:15:48.521 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.520+0000 7fbf11ffb640 1 -- 172.21.15.52:0/900556622 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e1deafb-c667-4984-8939-c657168e85da", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf1c1d9c40 con 0x7fbf1c144ab0 2024-03-30T07:15:48.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.520+0000 7fbf11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e1deafb-c667-4984-8939-c657168e85da", "id": 4}] 2024-03-30T07:15:48.522 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.520+0000 7fbf11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:48.522 INFO:teuthology.orchestra.run.smithi052.stdout:4 2024-03-30T07:15:48.523 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 10 librados: watch_flush enter 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 10 librados: watch_flush exit 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 10 client.4134.objecter successfully canceled tick 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 10 monclient: shutdown 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 msgr2=0x7fbf1c0af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 0x7fbf1c0af020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf14033000 tx=0x7fbf14004a40 comp rx=0 tx=0).stop 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 shutdown_connections 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbf1c144ab0 0x7fbf1c0af020 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbf1c0b0d20 0x7fbf1c0ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 --2- 172.21.15.52:0/900556622 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbf1c0b0120 0x7fbf1c0ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.524 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 >> 172.21.15.52:0/900556622 conn(0x7fbf1c0b70b0 msgr2=0x7fbf1c0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:48.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 shutdown_connections 2024-03-30T07:15:48.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.521+0000 7fbf23a28640 1 -- 172.21.15.52:0/900556622 wait complete. 2024-03-30T07:15:48.525 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.522+0000 7fbf23a28640 1 librados: shutdown 2024-03-30T07:15:48.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 1c5fc6e0-c7f9-41df-b24d-6150802b1e66 5 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.631+0000 7f98cc470640 10 monclient: get_monmap_and_config 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.631+0000 7f98cc470640 10 monclient: build_initial_monmap 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.631+0000 7f98cc470640 10 monclient: monmap: 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:48.633115+0000 2024-03-30T07:15:48.633 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:48.633115+0000 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.632+0000 7f98cc470640 1 Processor -- start 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.632+0000 7f98cc470640 1 -- start start 2024-03-30T07:15:48.634 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.632+0000 7f98cc470640 10 monclient: init 2024-03-30T07:15:48.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:48.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-03-30T07:15:48.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f98c40b0120 0x7f98c40a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-a con 0x7f98c40b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 0x7f98c40712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-b con 0x7f98c4144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98c40b0d20 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40a8bd0 con 0x7f98c40b0d20 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40aa4f0 con 0x7f98c40b0120 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4144f10 con 0x7f98c4144ab0 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98cc470640 10 monclient(hunting): authenticate will time out at 1484.779258s 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98caa0d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f98c40b0d20 0x7f98c41415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37166/0 (socket says 172.21.15.52:37166) 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98caa0d640 1 -- 172.21.15.52:0/2465520989 learned_addr learned my addr 172.21.15.52:0/2465520989 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.633+0000 7f98ca20c640 1 --2- 172.21.15.52:0/2465520989 >> v2:172.21.15.47:3300/0 conn(0x7f98c40b0120 0x7f98c40a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.637 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 1 --2- 172.21.15.52:0/2465520989 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 0x7f98c40712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): get_auth_request con 0x7f98c4144ab0 auth_method 0 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): get_auth_request con 0x7f98c40b0120 auth_method 0 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.634+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.635+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 1 -- 172.21.15.52:0/2465520989 >> v2:172.21.15.47:3300/0 conn(0x7f98c40b0120 msgr2=0x7f98c40a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 1 --2- 172.21.15.52:0/2465520989 >> v2:172.21.15.47:3300/0 conn(0x7f98c40b0120 0x7f98c40a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 1 -- 172.21.15.52:0/2465520989 >> v1:172.21.15.52:6789/0 conn(0x7f98c40b0d20 legacy=0x7f98c41415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 10 monclient: _finish_hunting 0 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 1 monclient: found mon.noname-b 2024-03-30T07:15:48.639 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:48.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.636+0000 7f98c9a0b640 1 -- 172.21.15.52:0/2465520989 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40b3780 con 0x7f98c4144ab0 2024-03-30T07:15:48.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.638+0000 7f98c9a0b640 1 --2- 172.21.15.52:0/2465520989 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 0x7f98c40712f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f98c00097a0 tx=0x7f98c002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb2ce998f6889498 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 1 -- 172.21.15.52:0/2465520989 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0031830 con 0x7f98c4144ab0 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: dump: 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.643 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: _finish_auth 0 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: _check_auth_tickets 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 1 -- 172.21.15.52:0/2465520989 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0031990 con 0x7f98c4144ab0 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 1 -- 172.21.15.52:0/2465520989 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c0031d20 con 0x7f98c4144ab0 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98cc470640 5 monclient: authenticate success, global_id 4127 2024-03-30T07:15:48.644 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98cc470640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98cc470640 4 set_mon_vals no callback set 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.642+0000 7f98c920a640 10 monclient: dump: 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.645 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 10 monclient: shutdown 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 -- 172.21.15.52:0/2465520989 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 msgr2=0x7f98c40712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 --2- 172.21.15.52:0/2465520989 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 0x7f98c40712f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f98c00097a0 tx=0x7f98c002b8d0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 -- 172.21.15.52:0/2465520989 shutdown_connections 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 --2- 172.21.15.52:0/2465520989 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c4144ab0 0x7f98c40712f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 --2- 172.21.15.52:0/2465520989 >> v2:172.21.15.47:3300/0 conn(0x7f98c40b0120 0x7f98c40a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 -- 172.21.15.52:0/2465520989 >> 172.21.15.52:0/2465520989 conn(0x7f98c40b70b0 msgr2=0x7f98c40b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 -- 172.21.15.52:0/2465520989 shutdown_connections 2024-03-30T07:15:48.651 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 1 -- 172.21.15.52:0/2465520989 wait complete. 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 10 monclient: build_initial_monmap 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.643+0000 7f98cc470640 10 monclient: monmap: 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 librados: starting msgr at 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 librados: starting objecter 2024-03-30T07:15:48.652 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 Processor -- start 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 -- start start 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 librados: setting wanted keys 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 librados: calling monclient init 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient: init 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-c con 0x7f98c40b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c40b0d20 0x7f98c413f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-a con 0x7f98c40b0d20 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98c4144ab0 0x7f98c413f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): picked mon.noname-b con 0x7f98c4144ab0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c40ab5b0 con 0x7f98c40b0d20 2024-03-30T07:15:48.653 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c413fe30 con 0x7f98c4144ab0 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c41423a0 con 0x7f98c40b0120 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.644+0000 7f98cc470640 10 monclient(hunting): authenticate will time out at 1484.790748s 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98c9a0b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c40b0d20 0x7f98c413f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.52:40726/0 (socket says 172.21.15.52:40726) 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 1 -- 172.21.15.52:0/3781184696 learned_addr learned my addr 172.21.15.52:0/3781184696 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 10 monclient(hunting): get_auth_request con 0x7f98c40b0120 auth_method 0 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98ca20c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.654 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98c9a0b640 10 monclient(hunting): get_auth_request con 0x7f98c40b0d20 auth_method 0 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98c9a0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98c9a0b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.645+0000 7f98c9a0b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98c4144ab0 0x7f98c413f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98ca20c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98c9a0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): get_auth_request con 0x7f98c4144ab0 auth_method 0 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:48.655 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.646+0000 7f98caa0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 10 monclient(hunting): handle_auth_done global_id 4130 payload 824 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 -- 172.21.15.52:0/3781184696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98c4144ab0 msgr2=0x7f98c413f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98c4144ab0 0x7f98c413f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 -- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c40b0d20 msgr2=0x7f98c413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c40b0d20 0x7f98c413f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 10 monclient: _finish_hunting 0 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 monclient: found mon.noname-c 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.656 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.649+0000 7f98ca20c640 1 -- 172.21.15.52:0/3781184696 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009400 con 0x7f98c40b0120 2024-03-30T07:15:48.657 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.651+0000 7f98ca20c640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f98b400e9e0 tx=0x7f98b400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.651+0000 7f98baffd640 10 client.?.objecter ms_handle_connect 0x7f98c40b0120 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.651+0000 7f98baffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4009670 con 0x7f98c40b0120 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: dump: 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.658 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: _finish_auth 0 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b400c9d0 con 0x7f98c40b0120 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 5 monclient: authenticate success, global_id 4130 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 10 monclient: _renew_subs 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.659 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c40ab7b0 con 0x7f98c40b0120 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400cd80 con 0x7f98c40b0120 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98baffd640 10 monclient: dump: 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:48.660 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98c8a09640 4 set_mon_vals no callback set 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 10 monclient: _renew_subs 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4141b70 con 0x7f98c40b0120 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.655+0000 7f98cc470640 1 librados: init done 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.656+0000 7f98cc470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.656+0000 7f98cc470640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.656+0000 7f98cc470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.661 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.656+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c404e030 con 0x7f98c40b0120 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f98b4005c90 con 0x7f98c40b0120 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f98b40139f0 con 0x7f98c40b0120 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 10 client.4130.objecter ms_dispatch 0x7f98c4000b90 osd_map(7..7 src has 1..7) v4 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 3 client.4130.objecter handle_osd_map got epochs [7,7] > 0 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 3 client.4130.objecter handle_osd_map decoding full epoch 7 2024-03-30T07:15:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.659+0000 7f98baffd640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-30T07:15:48.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.761+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f98b4021060 con 0x7f98c40b0120 2024-03-30T07:15:48.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.761+0000 7f98baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:48.762 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.761+0000 7f98baffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:48.834 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.833+0000 7f98cc470640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1c5fc6e0-c7f9-41df-b24d-6150802b1e66", "id": 5}] 2024-03-30T07:15:48.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.833+0000 7f98cc470640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1c5fc6e0-c7f9-41df-b24d-6150802b1e66", "id": 5}] 2024-03-30T07:15:48.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.833+0000 7f98cc470640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:15:48.835 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.833+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1c5fc6e0-c7f9-41df-b24d-6150802b1e66", "id": 5} v 0) v1 -- 0x7f98c4144110 con 0x7f98c40b0120 2024-03-30T07:15:48.942 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.941+0000 7f98baffd640 1 -- 172.21.15.52:0/3781184696 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c5fc6e0-c7f9-41df-b24d-6150802b1e66", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f98b4017070 con 0x7f98c40b0120 2024-03-30T07:15:48.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.941+0000 7f98baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1c5fc6e0-c7f9-41df-b24d-6150802b1e66", "id": 5}] 2024-03-30T07:15:48.943 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.941+0000 7f98baffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:48.943 INFO:teuthology.orchestra.run.smithi052.stdout:5 2024-03-30T07:15:48.944 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 10 librados: watch_flush enter 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 10 librados: watch_flush exit 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 10 client.4130.objecter successfully canceled tick 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 10 monclient: shutdown 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 msgr2=0x7f98c40b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f98b400e9e0 tx=0x7f98b400eea0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 shutdown_connections 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f98c4144ab0 0x7f98c413f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f98c40b0d20 0x7f98c413f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 --2- 172.21.15.52:0/3781184696 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f98c40b0120 0x7f98c40b66a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.942+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 >> 172.21.15.52:0/3781184696 conn(0x7f98c40b70b0 msgr2=0x7f98c4143090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:48.945 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.943+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 shutdown_connections 2024-03-30T07:15:48.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.943+0000 7f98cc470640 1 -- 172.21.15.52:0/3781184696 wait complete. 2024-03-30T07:15:48.946 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:48.943+0000 7f98cc470640 1 librados: shutdown 2024-03-30T07:15:48.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 4aa56a54-cd05-4f40-b7e3-a841be97681c 6 2024-03-30T07:15:48.965 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:48.964+0000 7fec5ce14040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T07:15:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 10 monclient: get_monmap_and_config 2024-03-30T07:15:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 10 monclient: build_initial_monmap 2024-03-30T07:15:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 10 monclient: monmap: 2024-03-30T07:15:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:49.054 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:49.054263+0000 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:49.054263+0000 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 1 Processor -- start 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 1 -- start start 2024-03-30T07:15:49.055 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.053+0000 7fe4858d4640 10 monclient: init 2024-03-30T07:15:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4800b0120 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4800a84a0 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe480144ab0 0x7fe48014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe480144ab0 addr v2:172.21.15.47:3300/0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4800a8900 con 0x7fe4800b0120 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4800a8bd0 con 0x7fe4800a84a0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4800b1430 con 0x7fe480144ab0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe4858d4640 10 monclient(hunting): authenticate will time out at 1485.200294s 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe47f7fe640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe480144ab0 0x7fe48014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.059 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe47e7fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe4800a84a0 0x7fe4800b0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:37178/0 (socket says 172.21.15.52:37178) 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.054+0000 7fe47e7fc640 1 -- 172.21.15.52:0/83849883 learned_addr learned my addr 172.21.15.52:0/83849883 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.055+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request con 0x7fe480144ab0 auth_method 0 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.055+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.055+0000 7fe47f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.055+0000 7fe47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.056+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.056+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.056+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 1 -- 172.21.15.52:0/83849883 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342119619 0 0) 0x7fe4800a8bd0 con 0x7fe4800a84a0 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 10 monclient(hunting): _negotiate 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 10 monclient(hunting): my global_id is 4132 2024-03-30T07:15:49.060 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.057+0000 7fe47dffb640 1 -- 172.21.15.52:0/83849883 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe468003520 con 0x7fe4800a84a0 2024-03-30T07:15:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2024-03-30T07:15:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 1 -- 172.21.15.52:0/83849883 >> v1:172.21.15.52:6789/0 conn(0x7fe4800a84a0 legacy=0x7fe4800b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 1 -- 172.21.15.52:0/83849883 >> v1:172.21.15.47:6789/0 conn(0x7fe4800b0120 legacy=0x7fe4800712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.061 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:15:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 1 monclient: found mon.noname-a 2024-03-30T07:15:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.059+0000 7fe47f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.060+0000 7fe47f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.062 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.060+0000 7fe47f7fe640 1 -- 172.21.15.52:0/83849883 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe480144f00 con 0x7fe480144ab0 2024-03-30T07:15:49.063 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.063+0000 7fe47f7fe640 1 --2- 172.21.15.52:0/83849883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe48014ce30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f43a3614629d90f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 1 -- 172.21.15.52:0/83849883 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47802cd00 con 0x7fe480144ab0 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: dump: 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.070 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: _finish_auth 0 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: _check_auth_tickets 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 5 monclient: authenticate success, global_id 4143 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 1 -- 172.21.15.52:0/83849883 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47802ce60 con 0x7fe480144ab0 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 1 -- 172.21.15.52:0/83849883 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478034660 con 0x7fe480144ab0 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 4 set_mon_vals no callback set 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe47dffb640 10 monclient: dump: 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.071 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 10 monclient: shutdown 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 1 -- 172.21.15.52:0/83849883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 msgr2=0x7fe48014ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.068+0000 7fe4858d4640 1 --2- 172.21.15.52:0/83849883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe48014ce30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47802bf50 comp rx=0 tx=0).stop 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 1 -- 172.21.15.52:0/83849883 shutdown_connections 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 1 --2- 172.21.15.52:0/83849883 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe48014ce30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.072 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 1 -- 172.21.15.52:0/83849883 >> 172.21.15.52:0/83849883 conn(0x7fe4800b70b0 msgr2=0x7fe4800b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:49.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 1 -- 172.21.15.52:0/83849883 shutdown_connections 2024-03-30T07:15:49.073 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 1 -- 172.21.15.52:0/83849883 wait complete. 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 10 monclient: build_initial_monmap 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.069+0000 7fe4858d4640 10 monclient: monmap: 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:49.076 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 librados: starting msgr at 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 librados: starting objecter 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 Processor -- start 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 -- start start 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 librados: setting wanted keys 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 librados: calling monclient init 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient: init 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe4800a84a0 0x7fe4801d9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-c con 0x7fe4800a84a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4800b0120 0x7fe480142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-b con 0x7fe4800b0120 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:49.077 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): picked mon.noname-a con 0x7fe480144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801d9800 con 0x7fe480144ab0 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48013f8d0 con 0x7fe4800b0120 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48013fa90 con 0x7fe4800a84a0 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.070+0000 7fe4858d4640 10 monclient(hunting): authenticate will time out at 1485.216865s 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47e7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4800b0120 0x7fe480142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47effd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe4800a84a0 0x7fe4801d9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.52:51356/0 (socket says 172.21.15.52:51356) 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 1 -- 172.21.15.52:0/749515172 learned_addr learned my addr 172.21.15.52:0/749515172 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:49.078 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request con 0x7fe480144ab0 auth_method 0 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47e7fc640 10 monclient(hunting): get_auth_request con 0x7fe4800b0120 auth_method 0 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47effd640 10 monclient(hunting): get_auth_request con 0x7fe4800a84a0 auth_method 0 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.071+0000 7fe47effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.079 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.072+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 -- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe4800a84a0 msgr2=0x7fe4801d9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe4800a84a0 0x7fe4801d9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 -- 172.21.15.52:0/749515172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4800b0120 msgr2=0x7fe480142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4800b0120 0x7fe480142f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 monclient: found mon.noname-a 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.080 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.074+0000 7fe47f7fe640 1 -- 172.21.15.52:0/749515172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478009a70 con 0x7fe480144ab0 2024-03-30T07:15:49.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.077+0000 7fe47f7fe640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe478004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:49.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.077+0000 7fe467fff640 10 client.?.objecter ms_handle_connect 0x7fe480144ab0 2024-03-30T07:15:49.081 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.077+0000 7fe467fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47803d600 con 0x7fe480144ab0 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: dump: 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.082 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: _finish_auth 0 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: _check_auth_tickets 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47803d780 con 0x7fe480144ab0 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47803db30 con 0x7fe480144ab0 2024-03-30T07:15:49.083 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe467fff640 10 monclient: dump: 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 5 monclient: authenticate success, global_id 4149 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 10 monclient: _renew_subs 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.084 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4801da260 con 0x7fe480144ab0 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe47d7fa640 4 set_mon_vals no callback set 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 10 monclient: _renew_subs 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe480142230 con 0x7fe480144ab0 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.081+0000 7fe4858d4640 1 librados: init done 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.082+0000 7fe4858d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.082+0000 7fe4858d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.085 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.082+0000 7fe4858d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.082+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4801d9660 con 0x7fe480144ab0 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe47803b040 con 0x7fe480144ab0 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fe4780529d0 con 0x7fe480144ab0 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 10 client.4149.objecter ms_dispatch 0x7fe480000b90 osd_map(8..8 src has 1..8) v4 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2024-03-30T07:15:49.086 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2024-03-30T07:15:49.087 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.085+0000 7fe467fff640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-30T07:15:49.094 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:49.094+0000 7efc18c36040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T07:15:49.188 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.188+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7fe4801d9660 con 0x7fe480144ab0 2024-03-30T07:15:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.188+0000 7fe467fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.189 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.188+0000 7fe467fff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:49.253 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:49.252+0000 7fec5ce14040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T07:15:49.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.260+0000 7fe4858d4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4aa56a54-cd05-4f40-b7e3-a841be97681c", "id": 6}] 2024-03-30T07:15:49.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.260+0000 7fe4858d4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4aa56a54-cd05-4f40-b7e3-a841be97681c", "id": 6}] 2024-03-30T07:15:49.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.260+0000 7fe4858d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:49.262 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.260+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4aa56a54-cd05-4f40-b7e3-a841be97681c", "id": 6} v 0) v1 -- 0x7fe4800722d0 con 0x7fe480144ab0 2024-03-30T07:15:49.305 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.304+0000 7fe467fff640 1 -- 172.21.15.52:0/749515172 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4aa56a54-cd05-4f40-b7e3-a841be97681c", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe4800722d0 con 0x7fe480144ab0 2024-03-30T07:15:49.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.304+0000 7fe467fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4aa56a54-cd05-4f40-b7e3-a841be97681c", "id": 6}] 2024-03-30T07:15:49.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.304+0000 7fe467fff640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:49.306 INFO:teuthology.orchestra.run.smithi052.stdout:6 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 10 librados: watch_flush enter 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 10 librados: watch_flush exit 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 10 client.4149.objecter successfully canceled tick 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 10 monclient: shutdown 2024-03-30T07:15:49.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:49.307 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fec5ce14040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T07:15:49.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 msgr2=0x7fe480143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe478004a40 comp rx=0 tx=0).stop 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 shutdown_connections 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe480144ab0 0x7fe480143480 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe4800b0120 0x7fe480142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 --2- 172.21.15.52:0/749515172 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe4800a84a0 0x7fe4801d9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 >> 172.21.15.52:0/749515172 conn(0x7fe4800b70b0 msgr2=0x7fe48013ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:49.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 shutdown_connections 2024-03-30T07:15:49.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 -- 172.21.15.52:0/749515172 wait complete. 2024-03-30T07:15:49.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.305+0000 7fe4858d4640 1 librados: shutdown 2024-03-30T07:15:49.317 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 09d844bd-e98e-4d61-a0ad-3c711bfe0c32 7 2024-03-30T07:15:49.378 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:49.377+0000 7efc18c36040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T07:15:49.430 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:49.428+0000 7fec5ce14040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T07:15:49.431 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:49.431+0000 7efc18c36040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T07:15:49.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.459+0000 7f62f8c87640 10 monclient: get_monmap_and_config 2024-03-30T07:15:49.460 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.459+0000 7f62f8c87640 10 monclient: build_initial_monmap 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.459+0000 7f62f8c87640 10 monclient: monmap: 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:49.460913+0000 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:49.460913+0000 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:49.461 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.463 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.460+0000 7f62f8c87640 1 Processor -- start 2024-03-30T07:15:49.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.460+0000 7f62f8c87640 1 -- start start 2024-03-30T07:15:49.465 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.460+0000 7f62f8c87640 10 monclient: init 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f62f40b0120 0x7f62f40a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-c con 0x7f62f40b0120 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62f4144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62f40b0d20 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40a8bd0 con 0x7f62f40b0d20 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f4144f10 con 0x7f62f4144ab0 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f40aa4f0 con 0x7f62f40b0120 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f8c87640 10 monclient(hunting): authenticate will time out at 1485.607083s 2024-03-30T07:15:49.466 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f62f40b0120 0x7f62f40a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f62f40b0120 0x7f62f40a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45758/0 (socket says 172.21.15.52:45758) 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 1 -- 172.21.15.52:0/3911267105 learned_addr learned my addr 172.21.15.52:0/3911267105 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 10 monclient(hunting): get_auth_request con 0x7f62f40b0120 auth_method 0 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.461+0000 7f62f259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.462+0000 7f62f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.462+0000 7f62f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.467 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.462+0000 7f62f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 1 -- 172.21.15.52:0/3911267105 >> v1:172.21.15.52:6789/0 conn(0x7f62f4144ab0 legacy=0x7f62f40712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 1 -- 172.21.15.52:0/3911267105 >> v1:172.21.15.47:6789/0 conn(0x7f62f40b0d20 legacy=0x7f62f414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 10 monclient: _finish_hunting 0 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 1 monclient: found mon.noname-c 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.464+0000 7f62f259c640 1 -- 172.21.15.52:0/3911267105 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40b3780 con 0x7f62f40b0120 2024-03-30T07:15:49.468 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.466+0000 7f62f259c640 1 --2- 172.21.15.52:0/3911267105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0120 0x7f62f40a84a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78187b36be228655 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 1 -- 172.21.15.52:0/3911267105 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4031870 con 0x7f62f40b0120 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: dump: 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.472 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: _finish_auth 0 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: _check_auth_tickets 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f8c87640 5 monclient: authenticate success, global_id 4138 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 1 -- 172.21.15.52:0/3911267105 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e40319d0 con 0x7f62f40b0120 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f8c87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:15:49.473 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f8c87640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f8c87640 4 set_mon_vals no callback set 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 1 -- 172.21.15.52:0/3911267105 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4031d60 con 0x7f62f40b0120 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.470+0000 7f62f1d9b640 10 monclient: dump: 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.474 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 10 monclient: shutdown 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 -- 172.21.15.52:0/3911267105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0120 msgr2=0x7f62f40a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 --2- 172.21.15.52:0/3911267105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0120 0x7f62f40a84a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f62e4009870 tx=0x7f62e402b910 comp rx=0 tx=0).stop 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 -- 172.21.15.52:0/3911267105 shutdown_connections 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 --2- 172.21.15.52:0/3911267105 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0120 0x7f62f40a84a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 -- 172.21.15.52:0/3911267105 >> 172.21.15.52:0/3911267105 conn(0x7f62f40b70b0 msgr2=0x7f62f40b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 -- 172.21.15.52:0/3911267105 shutdown_connections 2024-03-30T07:15:49.475 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.471+0000 7f62f8c87640 1 -- 172.21.15.52:0/3911267105 wait complete. 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 10 monclient: build_initial_monmap 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 10 monclient: monmap: 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:49.478 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 librados: starting msgr at 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 librados: starting objecter 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 Processor -- start 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 -- start start 2024-03-30T07:15:49.479 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 librados: setting wanted keys 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 1 librados: calling monclient init 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.472+0000 7f62f8c87640 10 monclient: init 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f62f40b0120 0x7f62f40ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-c con 0x7f62f40b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-b con 0x7f62f40b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f4144ab0 0x7f62f413c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): picked mon.noname-a con 0x7f62f4144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40af440 con 0x7f62f4144ab0 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.480 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40ab840 con 0x7f62f40b0d20 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f40ae4f0 con 0x7f62f40b0120 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f8c87640 10 monclient(hunting): authenticate will time out at 1485.619266s 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62ebfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.473+0000 7f62f2d9d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f4144ab0 0x7f62f413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.52:45766/0 (socket says 172.21.15.52:45766) 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 1 -- 172.21.15.52:0/4240581635 learned_addr learned my addr 172.21.15.52:0/4240581635 (peer_addr_for_me v2:172.21.15.52:0/0) 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): get_auth_request con 0x7f62f40b0d20 auth_method 0 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): get_auth_request con 0x7f62f4144ab0 auth_method 0 2024-03-30T07:15:49.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.474+0000 7f62f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 -- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f62f40b0120 msgr2=0x7f62f40ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f62f40b0120 0x7f62f40ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 -- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f4144ab0 msgr2=0x7f62f413c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f4144ab0 0x7f62f413c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:49.482 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 10 monclient: _finish_hunting 0 2024-03-30T07:15:49.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 monclient: found mon.noname-b 2024-03-30T07:15:49.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:49.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.483 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.477+0000 7f62ebfff640 1 -- 172.21.15.52:0/4240581635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e40094d0 con 0x7f62f40b0d20 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.479+0000 7f62ebfff640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00b3e0 tx=0x7f62dc00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.479+0000 7f62eaffd640 10 client.?.objecter ms_handle_connect 0x7f62f40b0d20 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.479+0000 7f62eaffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc011830 con 0x7f62f40b0d20 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: dump: 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.485 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.486 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.486 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.486 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: _finish_auth 0 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62dc0119b0 con 0x7f62f40b0d20 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 5 monclient: authenticate success, global_id 4141 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 10 monclient: _renew_subs 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f40ad1c0 con 0x7f62f40b0d20 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc011d60 con 0x7f62f40b0d20 2024-03-30T07:15:49.487 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62eaffd640 10 monclient: dump: 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f159a640 4 set_mon_vals no callback set 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:49.488 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 10 monclient: _renew_subs 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f413c990 con 0x7f62f40b0d20 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.483+0000 7f62f8c87640 1 librados: init done 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.484+0000 7f62f8c87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.484+0000 7f62f8c87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.484+0000 7f62f8c87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.489 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.484+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f41d66a0 con 0x7f62f40b0d20 2024-03-30T07:15:49.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f62dc024740 con 0x7f62f40b0d20 2024-03-30T07:15:49.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f62dc0233e0 con 0x7f62f40b0d20 2024-03-30T07:15:49.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 10 client.4141.objecter ms_dispatch 0x7f62f4000b90 osd_map(9..9 src has 1..9) v4 2024-03-30T07:15:49.508 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 3 client.4141.objecter handle_osd_map got epochs [9,9] > 0 2024-03-30T07:15:49.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 3 client.4141.objecter handle_osd_map decoding full epoch 9 2024-03-30T07:15:49.509 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.507+0000 7f62eaffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-30T07:15:49.560 INFO:tasks.ceph.mgr.y.smithi047.stderr:2024-03-30T07:15:49.560+0000 7efc18c36040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T07:15:49.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.688+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f62dc039c20 con 0x7f62f40b0d20 2024-03-30T07:15:49.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.688+0000 7f62eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:49.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.688+0000 7f62eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.766+0000 7f62f8c87640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "09d844bd-e98e-4d61-a0ad-3c711bfe0c32", "id": 7}] 2024-03-30T07:15:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.766+0000 7f62f8c87640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "09d844bd-e98e-4d61-a0ad-3c711bfe0c32", "id": 7}] 2024-03-30T07:15:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.766+0000 7f62f8c87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:49.767 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.766+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd new", "uuid": "09d844bd-e98e-4d61-a0ad-3c711bfe0c32", "id": 7} v 0) v1 -- 0x7f62f404e030 con 0x7f62f40b0d20 2024-03-30T07:15:49.940 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:49.939+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7f62dc0249f0 con 0x7f62f40b0d20 2024-03-30T07:15:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.318+0000 7f62eaffd640 1 -- 172.21.15.52:0/4240581635 <== mon.1 v2:172.21.15.52:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "09d844bd-e98e-4d61-a0ad-3c711bfe0c32", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f62dc021d60 con 0x7f62f40b0d20 2024-03-30T07:15:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.318+0000 7f62eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "09d844bd-e98e-4d61-a0ad-3c711bfe0c32", "id": 7}] 2024-03-30T07:15:50.320 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.318+0000 7f62eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:50.320 INFO:teuthology.orchestra.run.smithi052.stdout:7 2024-03-30T07:15:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.319+0000 7f62f8c87640 10 librados: watch_flush enter 2024-03-30T07:15:50.321 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 10 librados: watch_flush exit 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 10 client.4141.objecter successfully canceled tick 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 10 monclient: shutdown 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 msgr2=0x7f62f413be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00b3e0 tx=0x7f62dc00b8a0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 shutdown_connections 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f62f4144ab0 0x7f62f413c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f62f40b0d20 0x7f62f413be30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 --2- 172.21.15.52:0/4240581635 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f62f40b0120 0x7f62f40ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 >> 172.21.15.52:0/4240581635 conn(0x7f62f40b70b0 msgr2=0x7f62f4068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 shutdown_connections 2024-03-30T07:15:50.322 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 -- 172.21.15.52:0/4240581635 wait complete. 2024-03-30T07:15:50.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-30T07:15:50.320+0000 7f62f8c87640 1 librados: shutdown 2024-03-30T07:15:50.331 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.332 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-30T07:15:50.332 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-30T07:15:50.334 INFO:tasks.ceph.osd.0:Started 2024-03-30T07:15:50.334 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.334 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T07:15:50.334 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-30T07:15:50.336 INFO:tasks.ceph.osd.1:Started 2024-03-30T07:15:50.337 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.337 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-30T07:15:50.337 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-30T07:15:50.339 INFO:tasks.ceph.osd.2:Started 2024-03-30T07:15:50.339 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.339 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T07:15:50.339 DEBUG:teuthology.orchestra.run.smithi047:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-30T07:15:50.341 INFO:tasks.ceph.osd.3:Started 2024-03-30T07:15:50.341 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.341 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T07:15:50.341 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-30T07:15:50.343 INFO:tasks.ceph.osd.4:Started 2024-03-30T07:15:50.344 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.344 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T07:15:50.344 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-30T07:15:50.347 INFO:tasks.ceph.osd.5:Started 2024-03-30T07:15:50.347 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.347 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T07:15:50.347 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-30T07:15:50.349 INFO:tasks.ceph.osd.6:Started 2024-03-30T07:15:50.349 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-30T07:15:50.349 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T07:15:50.349 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-30T07:15:50.351 INFO:tasks.ceph.osd.7:Started 2024-03-30T07:15:50.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T07:15:50.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.509+0000 7f21124d4640 10 monclient: get_monmap_and_config 2024-03-30T07:15:50.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.509+0000 7f21124d4640 10 monclient: build_initial_monmap 2024-03-30T07:15:50.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.509+0000 7f21124d4640 10 monclient: monmap: 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:50.511473+0000 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:50.511473+0000 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:50.512 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.510+0000 7f21124d4640 1 Processor -- start 2024-03-30T07:15:50.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.510+0000 7f21124d4640 1 -- start start 2024-03-30T07:15:50.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.510+0000 7f21124d4640 10 monclient: init 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f210c0b4280 addr v1:172.21.15.52:6789/0 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f210c13e930 0x7f210c13eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-c con 0x7f210c13e930 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f210c141670 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c13f2e0 con 0x7f210c141670 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c0aa3c0 con 0x7f210c0b4280 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c0b4660 con 0x7f210c13e930 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:50.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.512+0000 7f21124d4640 10 monclient(hunting): authenticate will time out at 1476.628261s 2024-03-30T07:15:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.544+0000 7f2110a71640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f210c141670 0x7f210c141a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38554/0 (socket says 172.21.15.47:38554) 2024-03-30T07:15:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.544+0000 7f2110a71640 1 -- 172.21.15.47:0/1117494519 learned_addr learned my addr 172.21.15.47:0/1117494519 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:15:50.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.568+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1117494519 >> v2:172.21.15.52:3300/0 conn(0x7f210c13e930 0x7f210c13eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.568+0000 7f210b7fe640 10 monclient(hunting): get_auth_request con 0x7f210c13e930 auth_method 0 2024-03-30T07:15:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.569+0000 7f210b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.569+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.569+0000 7f210b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816185563 0 0) 0x7f210c13f2e0 con 0x7f210c141670 2024-03-30T07:15:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 10 monclient(hunting): _negotiate 2024-03-30T07:15:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 10 monclient(hunting): my global_id is 4155 2024-03-30T07:15:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.570+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4003520 con 0x7f210c141670 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004993630 0 0) 0x7f20f4003520 con 0x7f210c141670 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 >> v2:172.21.15.52:3300/0 conn(0x7f210c13e930 msgr2=0x7f210c13eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 --2- 172.21.15.47:0/1117494519 >> v2:172.21.15.52:3300/0 conn(0x7f210c13e930 0x7f210c13eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 >> v1:172.21.15.52:6789/0 conn(0x7f210c0b4280 legacy=0x7f210c0a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _finish_hunting 0 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 monclient: found mon.noname-a-legacy 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.47:6789/0 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c0aa630 con 0x7f210c141670 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _finish_auth 0 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:50.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f2104001a10 con 0x7f210c141670 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: dump: 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] but i'm connected to v1:172.21.15.47:6789/0, reconnecting 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 >> v1:172.21.15.47:6789/0 conn(0x7f210c141670 legacy=0x7f210c141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20f40039c0 0x7f20f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): picked mon.b con 0x7f20f40039c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 0x7f20f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): picked mon.a con 0x7f20f40047c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20f4005250 0x7f20f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): picked mon.c con 0x7f20f4005250 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0b4660 con 0x7f20f40047c0 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400c1a0 con 0x7f20f40039c0 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400c360 con 0x7f20f4005250 2024-03-30T07:15:50.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:50.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.576+0000 7f210affd640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:15:50.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.578+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 0x7f20f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:50.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.579+0000 7f210bfff640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20f40039c0 0x7f20f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:50.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.580+0000 7f210b7fe640 10 monclient(hunting): get_auth_request con 0x7f20f40047c0 auth_method 0 2024-03-30T07:15:50.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.580+0000 7f210b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.580+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.580+0000 7f210b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.581+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210bfff640 10 monclient(hunting): get_auth_request con 0x7f20f40039c0 auth_method 0 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.582+0000 7f210bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.584+0000 7f210bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.584+0000 7f210bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:50.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.584+0000 7f210bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:15:50.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 10 monclient(hunting): handle_auth_done global_id 4155 payload 293 2024-03-30T07:15:50.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 -- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20f4005250 msgr2=0x7f20f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20f4005250 0x7f20f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 -- 172.21.15.47:0/1117494519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20f40039c0 msgr2=0x7f20f4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20f40039c0 0x7f20f4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f2110a71640 1 -- 172.21.15.47:0/1117494519 reap_dead start 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 monclient: found mon.a 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 1 -- 172.21.15.47:0/1117494519 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4003520 con 0x7f20f40047c0 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 10 monclient: _finish_auth 0 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.587+0000 7f210b7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.588+0000 7f210b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.588+0000 7f21124d4640 5 monclient: authenticate success, global_id 4155 2024-03-30T07:15:50.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.588+0000 7f21124d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:15:50.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.593+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 0x7f20f4004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f210000ba10 tx=0x7f20f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0a7bd4a266de8c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100032630 con 0x7f20f40047c0 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 10 monclient: dump: 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:50.602 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f21124d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:15:50.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 1 -- 172.21.15.47:0/1117494519 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2100032790 con 0x7f20f40047c0 2024-03-30T07:15:50.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f210affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f21124d4640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.600+0000 7f21124d4640 4 set_mon_vals no callback set 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 10 monclient: shutdown 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 1 -- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 msgr2=0x7f20f4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 0x7f20f4004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f210000ba10 tx=0x7f20f40057f0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 1 -- 172.21.15.47:0/1117494519 shutdown_connections 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 1 --2- 172.21.15.47:0/1117494519 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20f40047c0 0x7f20f4004c40 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f210000ba10 tx=0x7f20f40057f0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.601+0000 7f21124d4640 1 -- 172.21.15.47:0/1117494519 >> 172.21.15.47:0/1117494519 conn(0x7f210c065e00 msgr2=0x7f210c066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.602+0000 7f21124d4640 1 -- 172.21.15.47:0/1117494519 shutdown_connections 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.602+0000 7f21124d4640 1 -- 172.21.15.47:0/1117494519 wait complete. 2024-03-30T07:15:50.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 10 monclient: build_initial_monmap 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 10 monclient: monmap: 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 librados: starting msgr at 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 librados: starting objecter 2024-03-30T07:15:50.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 Processor -- start 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 -- start start 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 librados: setting wanted keys 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 1 librados: calling monclient init 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.603+0000 7f21124d4640 10 monclient: init 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f210c1de090 0x7f210c1de470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-c con 0x7f210c1de090 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-b con 0x7f210c143da0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f210c1dafb0 0x7f210c1db430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): picked mon.noname-a con 0x7f210c1dafb0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c143a50 con 0x7f210c1dafb0 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c1de9b0 con 0x7f210c143da0 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c145510 con 0x7f210c1de090 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.604+0000 7f21124d4640 10 monclient(hunting): authenticate will time out at 1476.720949s 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:46318/0 (socket says 172.21.15.47:46318) 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 1 -- 172.21.15.47:0/1173700216 learned_addr learned my addr 172.21.15.47:0/1173700216 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 10 monclient(hunting): get_auth_request con 0x7f210c143da0 auth_method 0 2024-03-30T07:15:50.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f210b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.606+0000 7f2110a71640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f210c1dafb0 0x7f210c1db430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:50.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f210b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f2110a71640 10 monclient(hunting): get_auth_request con 0x7f210c1dafb0 auth_method 0 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f2110a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f2110a71640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.607+0000 7f2110a71640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.608+0000 7f2110a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.608+0000 7f2110a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:50.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.608+0000 7f2110a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 -- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f210c1de090 msgr2=0x7f210c1de470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f210c1de090 0x7f210c1de470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 -- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f210c1dafb0 msgr2=0x7f210c1db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f210c1dafb0 0x7f210c1db430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 monclient: found mon.noname-b 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:15:50.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.612+0000 7f210b7fe640 1 -- 172.21.15.47:0/1173700216 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c145780 con 0x7f210c143da0 2024-03-30T07:15:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.616+0000 7f210b7fe640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f210002bf80 tx=0x7f2100004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.616+0000 7f2108ff9640 10 client.?.objecter ms_handle_connect 0x7f210c143da0 2024-03-30T07:15:50.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.616+0000 7f2108ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210003ce40 con 0x7f210c143da0 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 10 monclient: dump: 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:50.622 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 10 monclient: _finish_auth 0 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f2108ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 5 monclient: authenticate success, global_id 4162 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 10 monclient: _renew_subs 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c143530 con 0x7f210c143da0 2024-03-30T07:15:50.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 10 monclient: _renew_subs 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c13c6d0 con 0x7f210c143da0 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.619+0000 7f21124d4640 1 librados: init done 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.620+0000 7f21124d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.620+0000 7f21124d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.620+0000 7f21124d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:50.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c04e030 con 0x7f210c143da0 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2100032e30 con 0x7f210c143da0 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100032ab0 con 0x7f210c143da0 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f2108ff9640 10 monclient: dump: 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:15:50.625 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:15:50.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.621+0000 7f210a7fc640 4 set_mon_vals no callback set 2024-03-30T07:15:50.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.641+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 2) v1 ==== 44528+0+0 (secure 0 0 0) 0x7f2100030400 con 0x7f210c143da0 2024-03-30T07:15:50.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f210002fd30 con 0x7f210c143da0 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 10 client.4162.objecter ms_dispatch 0x7f210c000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 3 client.4162.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 3 client.4162.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72706 (secure 0 0 0) 0x7f2100045350 con 0x7f210c143da0 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:50.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.825+0000 7f2108ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:50.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.947+0000 7f21124d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:50.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.947+0000 7f21124d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:50.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.947+0000 7f21124d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:15:50.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:50.948+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f210c0b5380 con 0x7f210c143da0 2024-03-30T07:15:51.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.197+0000 7f2108ff9640 1 -- 172.21.15.47:0/1173700216 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2100036030 con 0x7f210c143da0 2024-03-30T07:15:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.197+0000 7f2108ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.197+0000 7f2108ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:51.199 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:15:51.199 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:15:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 10 librados: watch_flush enter 2024-03-30T07:15:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 10 librados: watch_flush exit 2024-03-30T07:15:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 10 client.4162.objecter successfully canceled tick 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 10 monclient: shutdown 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 msgr2=0x7f210c1daa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f210002bf80 tx=0x7f2100004730 comp rx=0 tx=0).stop 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 shutdown_connections 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f210c1dafb0 0x7f210c1db430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f210c143da0 0x7f210c1daa70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 --2- 172.21.15.47:0/1173700216 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f210c1de090 0x7f210c1de470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 >> 172.21.15.47:0/1173700216 conn(0x7f210c065e00 msgr2=0x7f210c0b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 shutdown_connections 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 -- 172.21.15.47:0/1173700216 wait complete. 2024-03-30T07:15:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:15:51.198+0000 7f21124d4640 1 librados: shutdown 2024-03-30T07:15:51.215 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-30T07:15:51.216 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-30T07:15:51.680 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:51.679+0000 7febfd942640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-30T07:15:51.928 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-03-30T07:15:51.927+0000 7febfd942640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-30T07:15:52.021 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.164+0000 7f1a727e2640 10 monclient: get_monmap_and_config 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.164+0000 7f1a727e2640 10 monclient: build_initial_monmap 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.164+0000 7f1a727e2640 10 monclient: monmap: 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:52.165934+0000 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:52.165934+0000 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:15:52.166 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:15:52.167 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:15:52.167 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:15:52.167 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.165+0000 7f1a727e2640 1 Processor -- start 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.165+0000 7f1a727e2640 1 -- start start 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.165+0000 7f1a727e2640 10 monclient: init 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c0b43c0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1a6c13e790 0x7f1a6c13eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c13e790 addr v2:172.21.15.52:3300/0 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a6c141670 addr v1:172.21.15.47:6789/0 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c1421c0 con 0x7f1a6c141670 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0b47a0 con 0x7f1a6c13e790 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:52.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0a95a0 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.166+0000 7f1a727e2640 10 monclient(hunting): authenticate will time out at 1478.282595s 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:45484/0 (socket says 172.21.15.47:45484) 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 1 -- 172.21.15.47:0/3143520642 learned_addr learned my addr 172.21.15.47:0/3143520642 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request con 0x7f1a6c0b43c0 auth_method 0 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.167+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:52.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.168+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:52.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.168+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:52.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.168+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 1 -- 172.21.15.47:0/3143520642 >> v2:172.21.15.52:3300/0 conn(0x7f1a6c13e790 msgr2=0x7f1a6c13eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 1 --2- 172.21.15.47:0/3143520642 >> v2:172.21.15.52:3300/0 conn(0x7f1a6c13e790 0x7f1a6c13eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 1 -- 172.21.15.47:0/3143520642 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c141670 legacy=0x7f1a6c141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 10 monclient: _finish_hunting 0 2024-03-30T07:15:52.173 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 1 monclient: found mon.noname-b 2024-03-30T07:15:52.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:52.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:15:52.174 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.171+0000 7f1a6bfff640 1 -- 172.21.15.47:0/3143520642 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0aa350 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.174+0000 7f1a6bfff640 1 --2- 172.21.15.47:0/3143520642 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009870 tx=0x7f1a5c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d29145a334bfa4a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:52.293 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 1 -- 172.21.15.47:0/3143520642 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c0317c0 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: dump: 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:52.294 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: _finish_auth 0 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: _check_auth_tickets 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 1 -- 172.21.15.47:0/3143520642 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c031920 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 1 -- 172.21.15.47:0/3143520642 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c031cb0 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a6affd640 10 monclient: dump: 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.295 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a727e2640 5 monclient: authenticate success, global_id 4145 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a727e2640 10 monclient: get_monmap_and_config success 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.291+0000 7f1a727e2640 4 set_mon_vals no callback set 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 10 monclient: shutdown 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 -- 172.21.15.47:0/3143520642 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 msgr2=0x7f1a6c0a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 --2- 172.21.15.47:0/3143520642 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009870 tx=0x7f1a5c02b860 comp rx=0 tx=0).stop 2024-03-30T07:15:52.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 -- 172.21.15.47:0/3143520642 shutdown_connections 2024-03-30T07:15:52.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 --2- 172.21.15.47:0/3143520642 >> v2:172.21.15.52:3300/0 conn(0x7f1a6c13e790 0x7f1a6c13eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:52.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 --2- 172.21.15.47:0/3143520642 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c0b43c0 0x7f1a6c0a8990 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:52.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 -- 172.21.15.47:0/3143520642 >> 172.21.15.47:0/3143520642 conn(0x7f1a6c065f70 msgr2=0x7f1a6c066380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:52.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 -- 172.21.15.47:0/3143520642 shutdown_connections 2024-03-30T07:15:52.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.292+0000 7f1a727e2640 1 -- 172.21.15.47:0/3143520642 wait complete. 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.293+0000 7f1a727e2640 10 monclient: build_initial_monmap 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.293+0000 7f1a727e2640 10 monclient: monmap: 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:15:52.299 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.293+0000 7f1a727e2640 1 librados: starting msgr at 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.293+0000 7f1a727e2640 1 librados: starting objecter 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.293+0000 7f1a727e2640 1 Processor -- start 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.294+0000 7f1a727e2640 1 -- start start 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.294+0000 7f1a727e2640 1 librados: setting wanted keys 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.294+0000 7f1a727e2640 1 librados: calling monclient init 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.294+0000 7f1a727e2640 10 monclient: init 2024-03-30T07:15:52.300 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient: _reopen_session rank -1 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a6c0b43c0 0x7f1a6c13b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c0b43c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c13e790 0x7f1a6c0b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c13e790 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): picked mon.noname-a con 0x7f1a6c141670 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:52.301 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1337f0 con 0x7f1a6c141670 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1430a0 con 0x7f1a6c0b43c0 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c143310 con 0x7f1a6c13e790 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): _renew_subs 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a727e2640 10 monclient(hunting): authenticate will time out at 1478.411491s 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.295+0000 7f1a70d7f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a6bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a6c0b43c0 0x7f1a6c13b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51850/0 (socket says 172.21.15.47:51850) 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 1 -- 172.21.15.47:0/2081114553 learned_addr learned my addr 172.21.15.47:0/2081114553 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 10 monclient(hunting): get_auth_request con 0x7f1a6c141670 auth_method 0 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:52.302 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a70d7f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request con 0x7f1a6c0b43c0 auth_method 0 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.296+0000 7f1a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a70d7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a70d7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a70d7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.297+0000 7f1a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:15:52.303 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 10 monclient(hunting): handle_auth_done global_id 4161 payload 824 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 -- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c13e790 msgr2=0x7f1a6c0b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c13e790 0x7f1a6c0b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 -- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a6c0b43c0 msgr2=0x7f1a6c13b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a6c0b43c0 0x7f1a6c13b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 10 monclient: _finish_hunting 0 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 monclient: found mon.noname-a 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:15:52.304 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.301+0000 7f1a70d7f640 1 -- 172.21.15.47:0/2081114553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0094d0 con 0x7f1a6c141670 2024-03-30T07:15:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.304+0000 7f1a70d7f640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400b9c0 tx=0x7f1a6400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.304+0000 7f1a68ff9640 10 client.?.objecter ms_handle_connect 0x7f1a6c141670 2024-03-30T07:15:52.306 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.304+0000 7f1a68ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400cba0 con 0x7f1a6c141670 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: dump: 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:15:52.312 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: _finish_auth 0 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:15:52.313 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6400cd00 con 0x7f1a6c141670 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a64014680 con 0x7f1a6c141670 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a68ff9640 10 monclient: dump: 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:15:52.314 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a727e2640 5 monclient: authenticate success, global_id 4161 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a727e2640 10 monclient: _renew_subs 2024-03-30T07:15:52.315 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a727e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c0b7230 con 0x7f1a6c141670 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.310+0000 7f1a6a7fc640 4 set_mon_vals no callback set 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.311+0000 7f1a727e2640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.311+0000 7f1a727e2640 10 monclient: _renew_subs 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.311+0000 7f1a727e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.311+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c0b7cf0 con 0x7f1a6c141670 2024-03-30T07:15:52.316 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.311+0000 7f1a727e2640 1 librados: init done 2024-03-30T07:15:52.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.312+0000 7f1a727e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:52.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.312+0000 7f1a727e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:52.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.312+0000 7f1a727e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:52.317 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.312+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c0b7450 con 0x7f1a6c141670 2024-03-30T07:15:52.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.742+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 3) v1 ==== 88983+0+0 (secure 0 0 0) 0x7f1a6401d3b0 con 0x7f1a6c141670 2024-03-30T07:15:52.744 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.743+0000 7f1a68ff9640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 0x7f1a5806e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:15:52.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.743+0000 7f1a6bfff640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 0x7f1a5806e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:15:52.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.743+0000 7f1a6bfff640 10 monclient: get_auth_request con 0x7f1a5806c3e0 auth_method 0 2024-03-30T07:15:52.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.744+0000 7f1a68ff9640 10 client.4161.objecter ms_handle_connect 0x7f1a5806c3e0 2024-03-30T07:15:52.745 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.744+0000 7f1a6bfff640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 0x7f1a5806e8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c036000 tx=0x7f1a5c03d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:15:52.768 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f1a64058410 con 0x7f1a6c141670 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 10 client.4161.objecter ms_dispatch 0x7f1a6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 3 client.4161.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 3 client.4161.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1a64054750 con 0x7f1a6c141670 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:15:52.769 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:52.766+0000 7f1a68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:15:53.141 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.140+0000 7f1a727e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:53.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.140+0000 7f1a727e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:53.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.140+0000 7f1a727e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:15:53.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.140+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a6c0a9040 con 0x7f1a6c141670 2024-03-30T07:15:53.260 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.258+0000 7f1a68ff9640 1 -- 172.21.15.47:0/2081114553 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1a64019070 con 0x7f1a6c141670 2024-03-30T07:15:53.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.258+0000 7f1a68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:15:53.261 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.258+0000 7f1a68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:15:53.261 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:15:53.261 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 10 librados: watch_flush enter 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 10 librados: watch_flush exit 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 10 client.4161.objecter successfully canceled tick 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 msgr2=0x7f1a5806e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 0x7f1a5806e8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c036000 tx=0x7f1a5c03d040 comp rx=0 tx=0).stop 2024-03-30T07:15:53.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 10 monclient: shutdown 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 msgr2=0x7f1a6c0b5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400b9c0 tx=0x7f1a6400be80 comp rx=0 tx=0).stop 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 shutdown_connections 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a5806c3e0 0x7f1a5806e8a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a6c141670 0x7f1a6c0b5250 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a6c13e790 0x7f1a6c0b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 --2- 172.21.15.47:0/2081114553 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a6c0b43c0 0x7f1a6c13b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 >> 172.21.15.47:0/2081114553 conn(0x7f1a6c065f70 msgr2=0x7f1a6c0b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 shutdown_connections 2024-03-30T07:15:53.265 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 -- 172.21.15.47:0/2081114553 wait complete. 2024-03-30T07:15:53.266 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:15:53.261+0000 7f1a727e2640 1 librados: shutdown 2024-03-30T07:15:53.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T07:15:57.740 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:07.329 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:57.766 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:07.348 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:57.793 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:07.383 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:57.802 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:07.391 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:57.828 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:07.409 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:57.866 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:07.456 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.091 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:07.681 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.151 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:07.741 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.155 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:07.747 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.179 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:07.772 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.182 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:07.769 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.219 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:07.811 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.244 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:07.837 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.248 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:07.834 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.361 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:07.933 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.384 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:15:58.375+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.388 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:15:58.381+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.424 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:15:58.416+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.425 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:07.998 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.757 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:15:58.748+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.768 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:15:58.761+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.814 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:15:58.806+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.819 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:15:58.810+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:58.827 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:08.416 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.856 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:08.437 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:58.947 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:08.537 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.018 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:15:59.009+0000 5dc2300 -1 Falling back to public interface 2024-03-30T07:15:59.212 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:08.804 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.218 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:08.800 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.223 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:08.812 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.273 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:08.867 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.287 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:08.877 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.291 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:08.878 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.310 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:08.900 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.424 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.014 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.493 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.065 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.547 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.137 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.569 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.161 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.585 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:09.167 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.638 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.231 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.646 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:09.236 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.649 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.236 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.723 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.313 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.812 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.402 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.849 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.422 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.851 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:09.432 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.924 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.515 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.961 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:09.551 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.963 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.556 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:15:59.978 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.564 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.016 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.606 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.077 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.667 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.082 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:00.140 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:09.721 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.170+0000 7f5f1baae640 10 monclient: get_monmap_and_config 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.170+0000 7f5f1baae640 10 monclient: build_initial_monmap 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.170+0000 7f5f1baae640 10 monclient: monmap: 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:00.172187+0000 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:00.172187+0000 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:00.172 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:00.173 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:00.173 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:00.173 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:00.173 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 Processor -- start 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 -- start start 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient: init 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f5f140b4240 0x7f5f1413e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-c con 0x7f5f140b4240 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:00.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f1413eed0 0x7f5f14141560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1413eed0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f140a89f0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f140aa630 con 0x7f5f140a89f0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140b4620 con 0x7f5f140b4240 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140a8fa0 con 0x7f5f1413eed0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.171+0000 7f5f1baae640 10 monclient(hunting): authenticate will time out at 1486.288063s 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f1a04b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5f140a89f0 0x7f5f140b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36458/0 (socket says 172.21.15.47:36458) 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f1a04b640 1 -- 172.21.15.47:0/1273049328 learned_addr learned my addr 172.21.15.47:0/1273049328 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f19049640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f1413eed0 0x7f5f14141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f1984a640 1 --2- 172.21.15.47:0/1273049328 >> v2:172.21.15.52:3300/0 conn(0x7f5f140b4240 0x7f5f1413e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f19049640 10 monclient(hunting): get_auth_request con 0x7f5f1413eed0 auth_method 0 2024-03-30T07:16:00.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f19049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f19049640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.172+0000 7f5f19049640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): get_auth_request con 0x7f5f140b4240 auth_method 0 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:00.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.173+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:00.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 10 monclient(hunting): handle_auth_done global_id 4192 payload 274 2024-03-30T07:16:00.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f18848640 1 -- 172.21.15.47:0/1273049328 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634553065 0 0) 0x7f5f140aa630 con 0x7f5f140a89f0 2024-03-30T07:16:00.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 1 -- 172.21.15.47:0/1273049328 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f1413eed0 msgr2=0x7f5f14141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:00.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f1413eed0 0x7f5f14141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 1 -- 172.21.15.47:0/1273049328 >> v1:172.21.15.47:6789/0 conn(0x7f5f140a89f0 legacy=0x7f5f140b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 10 monclient: _finish_hunting 0 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 1 monclient: found mon.noname-c 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f1984a640 1 -- 172.21.15.47:0/1273049328 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f140b3980 con 0x7f5f140b4240 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.175+0000 7f5f18848640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-03-30T07:16:00.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.177+0000 7f5f1984a640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f140b4240 0x7f5f1413e990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f1002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3bea2d922a0992 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 1 -- 172.21.15.47:0/1273049328 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1002e070 con 0x7f5f140b4240 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: dump: 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:00.182 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: _finish_auth 0 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: _check_auth_tickets 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 1 -- 172.21.15.47:0/1273049328 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f100315a0 con 0x7f5f140b4240 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 1 -- 172.21.15.47:0/1273049328 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10031930 con 0x7f5f140b4240 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f1baae640 5 monclient: authenticate success, global_id 4192 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f1baae640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f1baae640 4 set_mon_vals no callback set 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:00.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.180+0000 7f5f18848640 10 monclient: dump: 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:00.184 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 10 monclient: shutdown 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 -- 172.21.15.47:0/1273049328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f140b4240 msgr2=0x7f5f1413e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f140b4240 0x7f5f1413e990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f100097a0 tx=0x7f5f1002b840 comp rx=0 tx=0).stop 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 -- 172.21.15.47:0/1273049328 shutdown_connections 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f1413eed0 0x7f5f14141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 --2- 172.21.15.47:0/1273049328 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f140b4240 0x7f5f1413e990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 -- 172.21.15.47:0/1273049328 >> 172.21.15.47:0/1273049328 conn(0x7f5f14065d30 msgr2=0x7f5f14066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 -- 172.21.15.47:0/1273049328 shutdown_connections 2024-03-30T07:16:00.188 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.181+0000 7f5f1baae640 1 -- 172.21.15.47:0/1273049328 wait complete. 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 10 monclient: build_initial_monmap 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 10 monclient: monmap: 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:00.189 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 librados: starting msgr at 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 librados: starting objecter 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 Processor -- start 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 -- start start 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 librados: setting wanted keys 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 1 librados: calling monclient init 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.182+0000 7f5f1baae640 10 monclient: init 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 0x7f5f1413b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-a con 0x7f5f140a89f0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:00.190 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 0x7f5f141d6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-c con 0x7f5f140b4240 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f1413eed0 0x7f5f141d6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1413eed0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f14138fa0 con 0x7f5f140a89f0 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1413b260 con 0x7f5f1413eed0 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f141d71a0 con 0x7f5f140b4240 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1baae640 10 monclient(hunting): authenticate will time out at 1486.299426s 2024-03-30T07:16:00.191 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f19049640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 0x7f5f141d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.183+0000 7f5f1984a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 0x7f5f1413b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 0x7f5f1413b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:34448/0 (socket says 172.21.15.47:34448) 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 1 -- 172.21.15.47:0/2069702306 learned_addr learned my addr 172.21.15.47:0/2069702306 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): get_auth_request con 0x7f5f140a89f0 auth_method 0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f1413eed0 0x7f5f141d6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): get_auth_request con 0x7f5f140b4240 auth_method 0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): get_auth_request con 0x7f5f1413eed0 auth_method 0 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:00.192 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f19049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:00.193 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.184+0000 7f5f1a04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 -- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f1413eed0 msgr2=0x7f5f141d6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f1413eed0 0x7f5f141d6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 -- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 msgr2=0x7f5f1413b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 0x7f5f1413b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.194 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 10 monclient: _finish_hunting 0 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 monclient: found mon.noname-c 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.187+0000 7f5f19049640 1 -- 172.21.15.47:0/2069702306 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10009400 con 0x7f5f140b4240 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.189+0000 7f5f19049640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 0x7f5f141d6720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400e840 tx=0x7f5f0400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.189+0000 7f5f0a7fc640 10 client.?.objecter ms_handle_connect 0x7f5f140b4240 2024-03-30T07:16:00.195 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.189+0000 7f5f0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:00.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400c730 con 0x7f5f140b4240 2024-03-30T07:16:00.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:00.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:00.196 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: dump: 2024-03-30T07:16:00.196 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0400c890 con 0x7f5f140b4240 2024-03-30T07:16:00.197 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400cc40 con 0x7f5f140b4240 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0a7fc640 10 monclient: dump: 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:00.198 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 5 monclient: authenticate success, global_id 4160 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 10 monclient: _renew_subs 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f0bfff640 4 set_mon_vals no callback set 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f140a9640 con 0x7f5f140b4240 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 10 monclient: _renew_subs 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f140b02b0 con 0x7f5f140b4240 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.193+0000 7f5f1baae640 1 librados: init done 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.194+0000 7f5f1baae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.194+0000 7f5f1baae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.194+0000 7f5f1baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:00.199 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.194+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f141d6440 con 0x7f5f140b4240 2024-03-30T07:16:00.216 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:09.808 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.219 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:09.792 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.229 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:09.819 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.231 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:09.824 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.244 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:09.830 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.305 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:09.895 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.343 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:09.933 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.429 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.010 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.455+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f5f04004020 con 0x7f5f140b4240 2024-03-30T07:16:00.457 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.456+0000 7f5f0a7fc640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 0x7f5ef806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:00.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.456+0000 7f5f1984a640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 0x7f5ef806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:00.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.456+0000 7f5f1984a640 10 monclient: get_auth_request con 0x7f5ef806c580 auth_method 0 2024-03-30T07:16:00.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.456+0000 7f5f1984a640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 0x7f5ef806ea40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f10000c00 tx=0x7f5f1003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:00.458 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.456+0000 7f5f0a7fc640 10 client.4160.objecter ms_handle_connect 0x7f5ef806c580 2024-03-30T07:16:00.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5f04057f70 con 0x7f5f140b4240 2024-03-30T07:16:00.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 10 client.4160.objecter ms_dispatch 0x7f5f14000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:16:00.469 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:16:00.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 3 client.4160.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:16:00.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 20 client.4160.objecter dump_active .. 0 homeless 2024-03-30T07:16:00.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f5f040542b0 con 0x7f5f140b4240 2024-03-30T07:16:00.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:00.470 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.468+0000 7f5f0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:00.494 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.087 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.495 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.084 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.515 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.107 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.518 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.104 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.520 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.093 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.594 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.184 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.612 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.201 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.651+0000 7f5f1baae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:00.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.651+0000 7f5f1baae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:00.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.651+0000 7f5f1baae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:00.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.651+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f141d6440 con 0x7f5f140b4240 2024-03-30T07:16:00.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.697+0000 7f5f0a7fc640 1 -- 172.21.15.47:0/2069702306 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5f141d6440 con 0x7f5f140b4240 2024-03-30T07:16:00.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.697+0000 7f5f0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:00.699 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.697+0000 7f5f0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:00.699 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:00.699 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:00.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 10 librados: watch_flush enter 2024-03-30T07:16:00.701 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 10 librados: watch_flush exit 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 20 client.4160.objecter shutdown clearing up homeless session... 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 10 client.4160.objecter successfully canceled tick 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 msgr2=0x7f5ef806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 0x7f5ef806ea40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f10000c00 tx=0x7f5f1003d040 comp rx=0 tx=0).stop 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 10 monclient: shutdown 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 msgr2=0x7f5f141d6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 0x7f5f141d6720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400e840 tx=0x7f5f0400ed00 comp rx=0 tx=0).stop 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 shutdown_connections 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f5ef806c580 0x7f5ef806ea40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f5f1413eed0 0x7f5f141d6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.702 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5f140b4240 0x7f5f141d6720 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 --2- 172.21.15.47:0/2069702306 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5f140a89f0 0x7f5f1413b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:00.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 >> 172.21.15.47:0/2069702306 conn(0x7f5f14065d30 msgr2=0x7f5f140b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:00.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 shutdown_connections 2024-03-30T07:16:00.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 -- 172.21.15.47:0/2069702306 wait complete. 2024-03-30T07:16:00.703 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:00.699+0000 7f5f1baae640 1 librados: shutdown 2024-03-30T07:16:00.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T07:16:00.721 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.302 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.762 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.355 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.762 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.352 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.790 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.376 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.812 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.403 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.816 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.389 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.879 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.469 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:00.884 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.475 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.011 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.592 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.028 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.618 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.030 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.623 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.064 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.650 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.116 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.708 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.123 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.696 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.146 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:10.736 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.179 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:10.769 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.299 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:10.892 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.299 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:10.889 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.307 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:10.888 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.334 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:10.920 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.405 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:10.997 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.420 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:10.993 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.564 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:11.157 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.576 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.166 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.598 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:11.184 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.624 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:11.213 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.680 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:11.269 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.704 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:11.295 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.711 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:11.284 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.711 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:11.301 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.767 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:11.357 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.837 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:11.418 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:01.893 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:11.474 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.010 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:11.583 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.053 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:11.647 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.087 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:11.674 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.110 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:11.703 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.128 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.716 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.165 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:11.751 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.182 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:11.772 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.250 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:11.841 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.309 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:11.901 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.322 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:11.894 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:02.378 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:11.950 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.207 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:16.788 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.322 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:16.911 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.457 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:17.047 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.516 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:07.518 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:17.108 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.647+0000 7f7b45bfb640 10 monclient: get_monmap_and_config 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.647+0000 7f7b45bfb640 10 monclient: build_initial_monmap 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.647+0000 7f7b45bfb640 10 monclient: monmap: 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:07.648003+0000 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:07.648003+0000 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:07.648 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:07.649 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:07.649 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:07.649 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:07.649 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 1 Processor -- start 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 1 -- start start 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient: init 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f7b400b42a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b40141b40 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:07.650 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b4013ef40 0x7f7b4014be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f7b4013ef40 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.648+0000 7f7b45bfb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400aa6a0 con 0x7f7b40141b40 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b40141fa0 con 0x7f7b4013ef40 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400b4680 con 0x7f7b400b42a0 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:07.651 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b45bfb640 10 monclient(hunting): authenticate will time out at 1493.764160s 2024-03-30T07:16:07.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3ffff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b4013ef40 0x7f7b4014be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:50126/0 (socket says 172.21.15.47:50126) 2024-03-30T07:16:07.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 1 -- 172.21.15.47:0/891066652 learned_addr learned my addr 172.21.15.47:0/891066652 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:07.652 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 10 monclient(hunting): get_auth_request con 0x7f7b400b42a0 auth_method 0 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.649+0000 7f7b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): get_auth_request con 0x7f7b4013ef40 auth_method 0 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:07.653 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.650+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:07.655 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 1 -- 172.21.15.47:0/891066652 >> v2:172.21.15.47:3300/0 conn(0x7f7b4013ef40 msgr2=0x7f7b4014be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 1 --2- 172.21.15.47:0/891066652 >> v2:172.21.15.47:3300/0 conn(0x7f7b4013ef40 0x7f7b4014be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 1 -- 172.21.15.47:0/891066652 >> v1:172.21.15.52:6789/0 conn(0x7f7b40141b40 legacy=0x7f7b4013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 1 monclient: found mon.noname-b 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.652+0000 7f7b3f7fe640 1 -- 172.21.15.47:0/891066652 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b400a8e70 con 0x7f7b400b42a0 2024-03-30T07:16:07.656 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.654+0000 7f7b3f7fe640 1 --2- 172.21.15.47:0/891066652 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68421290f29b441a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 1 -- 172.21.15.47:0/891066652 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30030840 con 0x7f7b400b42a0 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: dump: 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:07.659 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 1 -- 172.21.15.47:0/891066652 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b300309a0 con 0x7f7b400b42a0 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:07.660 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 1 -- 172.21.15.47:0/891066652 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30030d30 con 0x7f7b400b42a0 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b45bfb640 5 monclient: authenticate success, global_id 4163 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b45bfb640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b45bfb640 4 set_mon_vals no callback set 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.657+0000 7f7b3e7fc640 10 monclient: dump: 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:07.661 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 10 monclient: shutdown 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 -- 172.21.15.47:0/891066652 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 msgr2=0x7f7b40141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/891066652 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b3002b980 comp rx=0 tx=0).stop 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 -- 172.21.15.47:0/891066652 shutdown_connections 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/891066652 >> v2:172.21.15.47:3300/0 conn(0x7f7b4013ef40 0x7f7b4014be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/891066652 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40141600 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 -- 172.21.15.47:0/891066652 >> 172.21.15.47:0/891066652 conn(0x7f7b40065d30 msgr2=0x7f7b40066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 -- 172.21.15.47:0/891066652 shutdown_connections 2024-03-30T07:16:07.662 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.658+0000 7f7b45bfb640 1 -- 172.21.15.47:0/891066652 wait complete. 2024-03-30T07:16:07.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.659+0000 7f7b45bfb640 10 monclient: build_initial_monmap 2024-03-30T07:16:07.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.659+0000 7f7b45bfb640 10 monclient: monmap: 2024-03-30T07:16:07.666 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.659+0000 7f7b45bfb640 1 librados: starting msgr at 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.659+0000 7f7b45bfb640 1 librados: starting objecter 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 Processor -- start 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 -- start start 2024-03-30T07:16:07.667 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 librados: setting wanted keys 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 librados: calling monclient init 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient: init 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f7b400b42a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b4013ef40 0x7f7b40134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4013ef40 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f7b40141b40 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40142900 con 0x7f7b40141b40 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:07.668 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40142ba0 con 0x7f7b4013ef40 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40134ec0 con 0x7f7b400b42a0 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.660+0000 7f7b45bfb640 10 monclient(hunting): authenticate will time out at 1493.776044s 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3f7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33428/0 (socket says 172.21.15.47:33428) 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 1 -- 172.21.15.47:0/2369179214 learned_addr learned my addr 172.21.15.47:0/2369179214 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 10 monclient(hunting): get_auth_request con 0x7f7b40141b40 auth_method 0 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:07.669 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3f7fe640 10 monclient(hunting): get_auth_request con 0x7f7b400b42a0 auth_method 0 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.661+0000 7f7b3effd640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b4013ef40 0x7f7b40134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): get_auth_request con 0x7f7b4013ef40 auth_method 0 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:07.670 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.662+0000 7f7b3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.664+0000 7f7b3ffff640 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.664+0000 7f7b3ffff640 1 -- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 msgr2=0x7f7b40136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.664+0000 7f7b3ffff640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 1 -- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b4013ef40 msgr2=0x7f7b40134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b4013ef40 0x7f7b40134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 1 monclient: found mon.noname-a 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:07.671 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.665+0000 7f7b3ffff640 1 -- 172.21.15.47:0/2369179214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b300094d0 con 0x7f7b40141b40 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.667+0000 7f7b3ffff640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b38016710 tx=0x7f7b38016b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.667+0000 7f7b27fff640 10 client.?.objecter ms_handle_connect 0x7f7b40141b40 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.667+0000 7f7b27fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38018040 con 0x7f7b40141b40 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: dump: 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:07.674 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: _finish_auth 0 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b38014840 con 0x7f7b40141b40 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38009ca0 con 0x7f7b40141b40 2024-03-30T07:16:07.675 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b27fff640 10 monclient: dump: 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.671+0000 7f7b3dffb640 4 set_mon_vals no callback set 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 5 monclient: authenticate success, global_id 4194 2024-03-30T07:16:07.676 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 10 monclient: _renew_subs 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b400b5e50 con 0x7f7b40141b40 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 10 monclient: _renew_subs 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b40143310 con 0x7f7b40141b40 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.672+0000 7f7b45bfb640 1 librados: init done 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.673+0000 7f7b45bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.673+0000 7f7b45bfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.673+0000 7f7b45bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:07.677 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.673+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b40142d50 con 0x7f7b40141b40 2024-03-30T07:16:07.678 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:17.262 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:07.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.708+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f7b38014bf0 con 0x7f7b40141b40 2024-03-30T07:16:07.709 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.709+0000 7f7b27fff640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 0x7f7b2c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:07.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.709+0000 7f7b3f7fe640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 0x7f7b2c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:07.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.709+0000 7f7b3f7fe640 10 monclient: get_auth_request con 0x7f7b2c06c580 auth_method 0 2024-03-30T07:16:07.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.710+0000 7f7b3f7fe640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 0x7f7b2c06ea40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b30008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:07.710 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.710+0000 7f7b27fff640 10 client.4194.objecter ms_handle_connect 0x7f7b2c06c580 2024-03-30T07:16:07.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.714+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7b38061810 con 0x7f7b40141b40 2024-03-30T07:16:07.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.714+0000 7f7b27fff640 10 client.4194.objecter ms_dispatch 0x7f7b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:16:07.714 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.714+0000 7f7b27fff640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:16:07.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.714+0000 7f7b27fff640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:16:07.715 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:07.714+0000 7f7b27fff640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-30T07:16:07.957 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:17.529 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:08.029 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:17.622 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:08.131 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:17.722 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:08.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.130+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7b3805db50 con 0x7f7b40141b40 2024-03-30T07:16:08.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.130+0000 7f7b27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:08.131 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.130+0000 7f7b27fff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:08.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.354+0000 7f7b45bfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:08.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.354+0000 7f7b45bfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:08.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.354+0000 7f7b45bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:08.355 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.354+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b40142d50 con 0x7f7b40141b40 2024-03-30T07:16:08.377 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.377+0000 7f7b27fff640 1 -- 172.21.15.47:0/2369179214 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7b40142d50 con 0x7f7b40141b40 2024-03-30T07:16:08.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.377+0000 7f7b27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:08.378 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.377+0000 7f7b27fff640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:08.378 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:08.378 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 10 librados: watch_flush enter 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 10 librados: watch_flush exit 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 10 client.4194.objecter successfully canceled tick 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 msgr2=0x7f7b2c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 0x7f7b2c06ea40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009870 tx=0x7f7b30008030 comp rx=0 tx=0).stop 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 10 monclient: shutdown 2024-03-30T07:16:08.380 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 msgr2=0x7f7b40134980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b38016710 tx=0x7f7b38016b10 comp rx=0 tx=0).stop 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 shutdown_connections 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7b2c06c580 0x7f7b2c06ea40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b40141b40 0x7f7b40134980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:08.381 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7b4013ef40 0x7f7b40134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:08.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 --2- 172.21.15.47:0/2369179214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b400b42a0 0x7f7b40136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:08.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.379+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 >> 172.21.15.47:0/2369179214 conn(0x7f7b40065d30 msgr2=0x7f7b4013f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:08.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.380+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 shutdown_connections 2024-03-30T07:16:08.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.380+0000 7f7b45bfb640 1 -- 172.21.15.47:0/2369179214 wait complete. 2024-03-30T07:16:08.382 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:08.380+0000 7f7b45bfb640 1 librados: shutdown 2024-03-30T07:16:08.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T07:16:13.035 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:22.625 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.039 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:22.629 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.348 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:22.938 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.352 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:22.942 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.353 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:22.943 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.357 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:22.947 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.371 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:22.952 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.377 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:22.958 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.711 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:23.297 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:13.715 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:23.301 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.157 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:23.749 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.160 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:23.733 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.162 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:23.754 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.164 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:23.737 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.233 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:23.827 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:14.240 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:23.833 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:15.200 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.319+0000 7fd54e6c4640 10 monclient: get_monmap_and_config 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.319+0000 7fd54e6c4640 10 monclient: build_initial_monmap 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.319+0000 7fd54e6c4640 10 monclient: monmap: 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:15.320 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:15.320371+0000 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:15.320371+0000 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:15.321 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.320+0000 7fd54e6c4640 1 Processor -- start 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.320+0000 7fd54e6c4640 1 -- start start 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.320+0000 7fd54e6c4640 10 monclient: init 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5480b42a0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd548141d10 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd54814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:15.323 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd54813ef40 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480aa6a0 con 0x7fd5480b42a0 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480a8e70 con 0x7fd548141d10 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5480b4680 con 0x7fd54813ef40 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd54e6c4640 10 monclient(hunting): authenticate will time out at 1501.436418s 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd547fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd5480b42a0 0x7fd548141600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45478/0 (socket says 172.21.15.47:45478) 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.321+0000 7fd547fff640 1 -- 172.21.15.47:0/1861526740 learned_addr learned my addr 172.21.15.47:0/1861526740 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.322+0000 7fd54cc61640 1 --2- 172.21.15.47:0/1861526740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd54814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.322+0000 7fd54cc61640 10 monclient(hunting): get_auth_request con 0x7fd54813ef40 auth_method 0 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.322+0000 7fd54cc61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:15.324 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.322+0000 7fd54cc61640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:15.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.322+0000 7fd54cc61640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:15.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.323+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:15.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.323+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:15.325 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.323+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:15.326 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 1 -- 172.21.15.47:0/1861526740 >> v1:172.21.15.52:6789/0 conn(0x7fd548141d10 legacy=0x7fd54813e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 1 -- 172.21.15.47:0/1861526740 >> v1:172.21.15.47:6789/0 conn(0x7fd5480b42a0 legacy=0x7fd548141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 10 monclient: _finish_hunting 0 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 1 monclient: found mon.noname-b 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:15.327 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.325+0000 7fd54cc61640 1 -- 172.21.15.47:0/1861526740 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480b39f0 con 0x7fd54813ef40 2024-03-30T07:16:15.328 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.328+0000 7fd54cc61640 1 --2- 172.21.15.47:0/1861526740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd54814ce80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd540009dc0 tx=0x7fd54002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46e49abdf945443d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:15.332 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 1 -- 172.21.15.47:0/1861526740 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54002caa0 con 0x7fd54813ef40 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: dump: 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:15.333 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 1 -- 172.21.15.47:0/1861526740 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54002cc00 con 0x7fd54813ef40 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 1 -- 172.21.15.47:0/1861526740 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540034440 con 0x7fd54813ef40 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd546ffd640 10 monclient: dump: 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:15.334 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 5 monclient: authenticate success, global_id 4169 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:15.335 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 4 set_mon_vals no callback set 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 10 monclient: shutdown 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1861526740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 msgr2=0x7fd54814ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.332+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1861526740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd54814ce80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd540009dc0 tx=0x7fd54002bd70 comp rx=0 tx=0).stop 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1861526740 shutdown_connections 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1861526740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd54814ce80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1861526740 >> 172.21.15.47:0/1861526740 conn(0x7fd548065d30 msgr2=0x7fd548066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1861526740 shutdown_connections 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1861526740 wait complete. 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 10 monclient: build_initial_monmap 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.333+0000 7fd54e6c4640 10 monclient: monmap: 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:15.339 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 librados: starting msgr at 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 librados: starting objecter 2024-03-30T07:16:15.340 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 Processor -- start 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 -- start start 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 librados: setting wanted keys 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 1 librados: calling monclient init 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.334+0000 7fd54e6c4640 10 monclient: init 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5480b42a0 0x7fd5481d6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5480b42a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd54813ef40 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5481d71e0 0x7fd5481d9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd5481d71e0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5481d6530 con 0x7fd5481d71e0 2024-03-30T07:16:15.341 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5481d6780 con 0x7fd5480b42a0 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd548144760 con 0x7fd54813ef40 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54e6c4640 10 monclient(hunting): authenticate will time out at 1501.450439s 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd54cc61640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5481d71e0 0x7fd5481d9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.335+0000 7fd547fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5480b42a0 0x7fd5481d6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:50146/0 (socket says 172.21.15.47:50146) 2024-03-30T07:16:15.342 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 1 -- 172.21.15.47:0/1232430853 learned_addr learned my addr 172.21.15.47:0/1232430853 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): get_auth_request con 0x7fd54813ef40 auth_method 0 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): get_auth_request con 0x7fd5481d71e0 auth_method 0 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd547fff640 10 monclient(hunting): get_auth_request con 0x7fd5480b42a0 auth_method 0 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd547fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd547fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd547fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:15.343 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.336+0000 7fd54cc61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.337+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.337+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:15.344 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.337+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:15.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 10 monclient(hunting): handle_auth_done global_id 4172 payload 824 2024-03-30T07:16:15.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 -- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5480b42a0 msgr2=0x7fd5481d6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:15.345 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5480b42a0 0x7fd5481d6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 -- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5481d71e0 msgr2=0x7fd5481d9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5481d71e0 0x7fd5481d9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 10 monclient: _finish_hunting 0 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 monclient: found mon.noname-c 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.340+0000 7fd5477fe640 1 -- 172.21.15.47:0/1232430853 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540009a70 con 0x7fd54813ef40 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.343+0000 7fd5477fe640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c780 tx=0x7fd53c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.343+0000 7fd544ff9640 10 client.?.objecter ms_handle_connect 0x7fd54813ef40 2024-03-30T07:16:15.346 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.343+0000 7fd544ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c005270 con 0x7fd54813ef40 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: dump: 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.350 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: _finish_auth 0 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c0053d0 con 0x7fd54813ef40 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 5 monclient: authenticate success, global_id 4172 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 10 monclient: _renew_subs 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:15.351 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5480b5a60 con 0x7fd54813ef40 2024-03-30T07:16:15.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:15.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00a880 con 0x7fd54813ef40 2024-03-30T07:16:15.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:15.352 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd544ff9640 10 monclient: dump: 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 10 client.4172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 10 monclient: _renew_subs 2024-03-30T07:16:15.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5480b5740 con 0x7fd54813ef40 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd5467fc640 4 set_mon_vals no callback set 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.347+0000 7fd54e6c4640 1 librados: init done 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.348+0000 7fd54e6c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.348+0000 7fd54e6c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.348+0000 7fd54e6c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:15.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.348+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5480a8980 con 0x7fd54813ef40 2024-03-30T07:16:15.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.365+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fd53c00a9e0 con 0x7fd54813ef40 2024-03-30T07:16:15.366 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.366+0000 7fd544ff9640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 0x7fd53406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:15.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.366+0000 7fd547fff640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 0x7fd53406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:15.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.366+0000 7fd547fff640 10 monclient: get_auth_request con 0x7fd53406c550 auth_method 0 2024-03-30T07:16:15.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.367+0000 7fd547fff640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 0x7fd53406ea10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd538005880 tx=0x7fd538009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:15.367 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.367+0000 7fd544ff9640 10 client.4172.objecter ms_handle_connect 0x7fd53406c550 2024-03-30T07:16:15.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fd53c057dc0 con 0x7fd54813ef40 2024-03-30T07:16:15.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 10 client.4172.objecter ms_dispatch 0x7fd548000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:16:15.686 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 3 client.4172.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:16:15.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 3 client.4172.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:16:15.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 20 client.4172.objecter dump_active .. 0 homeless 2024-03-30T07:16:15.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fd53c054100 con 0x7fd54813ef40 2024-03-30T07:16:15.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:15.687 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:15.685+0000 7fd544ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:16.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.031+0000 7fd54e6c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:16.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.031+0000 7fd54e6c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:16.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.031+0000 7fd54e6c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:16.032 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.031+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd54804e0a0 con 0x7fd54813ef40 2024-03-30T07:16:16.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.171+0000 7fd544ff9640 1 -- 172.21.15.47:0/1232430853 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fd53c017020 con 0x7fd54813ef40 2024-03-30T07:16:16.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.171+0000 7fd544ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:16.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.171+0000 7fd544ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:16.172 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:16.172 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 10 librados: watch_flush enter 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 10 librados: watch_flush exit 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 20 client.4172.objecter shutdown clearing up homeless session... 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 10 client.4172.objecter successfully canceled tick 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 msgr2=0x7fd53406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 0x7fd53406ea10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd538005880 tx=0x7fd538009040 comp rx=0 tx=0).stop 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 10 monclient: shutdown 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 msgr2=0x7fd5480b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:16.175 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.174+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c780 tx=0x7fd53c00cc40 comp rx=0 tx=0).stop 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 shutdown_connections 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd53406c550 0x7fd53406ea10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5481d71e0 0x7fd5481d9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd54813ef40 0x7fd5480b5020 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 --2- 172.21.15.47:0/1232430853 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd5480b42a0 0x7fd5481d6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:16.176 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 >> 172.21.15.47:0/1232430853 conn(0x7fd548065d30 msgr2=0x7fd5480b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:16.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 shutdown_connections 2024-03-30T07:16:16.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 -- 172.21.15.47:0/1232430853 wait complete. 2024-03-30T07:16:16.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:16.175+0000 7fd54e6c4640 1 librados: shutdown 2024-03-30T07:16:16.190 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T07:16:17.076 INFO:tasks.ceph.osd.5.smithi052.stderr:==00:00:00:26.666 35314== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:17.485 INFO:tasks.ceph.osd.4.smithi052.stderr:==00:00:00:27.075 35313== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:17.602 INFO:tasks.ceph.osd.6.smithi052.stderr:==00:00:00:27.191 35315== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:17.612 INFO:tasks.ceph.osd.7.smithi052.stderr:==00:00:00:27.193 35316== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:18.058 INFO:tasks.ceph.osd.3.smithi047.stderr:==00:00:00:27.644 35206== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:18.523 INFO:tasks.ceph.osd.2.smithi047.stderr:==00:00:00:28.096 35207== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:18.709 INFO:tasks.ceph.osd.0.smithi047.stderr:==00:00:00:28.300 35205== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:18.758 INFO:tasks.ceph.osd.1.smithi047.stderr:==00:00:00:28.351 35204== Warning: unimplemented fcntl command: 1038 2024-03-30T07:16:22.998 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:23.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.141+0000 7fc3f2918640 10 monclient: get_monmap_and_config 2024-03-30T07:16:23.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.141+0000 7fc3f2918640 10 monclient: build_initial_monmap 2024-03-30T07:16:23.142 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.141+0000 7fc3f2918640 10 monclient: monmap: 2024-03-30T07:16:23.142 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:23.142 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:23.142411+0000 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:23.142411+0000 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:23.143 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.144 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.142+0000 7fc3f2918640 1 Processor -- start 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.142+0000 7fc3f2918640 1 -- start start 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.142+0000 7fc3f2918640 10 monclient: init 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc3ec0b42a0 0x7fc3ec141600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ec0b42a0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3ec141b40 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc3ec13ef40 0x7fc3ec14be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ec13ef40 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec0aa6a0 con 0x7fc3ec141b40 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec141fa0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:23.145 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec0b4680 con 0x7fc3ec13ef40 2024-03-30T07:16:23.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:23.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.143+0000 7fc3f2918640 10 monclient(hunting): authenticate will time out at 1509.259048s 2024-03-30T07:16:23.146 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.144+0000 7fc3eb7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc3ec141b40 0x7fc3ec13e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52782/0 (socket says 172.21.15.47:52782) 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.144+0000 7fc3eb7fe640 1 -- 172.21.15.47:0/785440147 learned_addr learned my addr 172.21.15.47:0/785440147 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.145+0000 7fc3ebfff640 1 --2- 172.21.15.47:0/785440147 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec0b42a0 0x7fc3ec141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.145+0000 7fc3f0eb5640 1 --2- 172.21.15.47:0/785440147 >> v2:172.21.15.52:3300/0 conn(0x7fc3ec13ef40 0x7fc3ec14be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.146+0000 7fc3f0eb5640 10 monclient(hunting): get_auth_request con 0x7fc3ec13ef40 auth_method 0 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.146+0000 7fc3f0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.146+0000 7fc3f0eb5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:23.147 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.146+0000 7fc3f0eb5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.148+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.148+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.148+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.149+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request con 0x7fc3ec0b42a0 auth_method 0 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.149+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.149+0000 7fc3ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:23.149 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.149+0000 7fc3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_done global_id 4210 payload 274 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 1 -- 172.21.15.47:0/785440147 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec0b42a0 msgr2=0x7fc3ec141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 1 --2- 172.21.15.47:0/785440147 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec0b42a0 0x7fc3ec141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 1 -- 172.21.15.47:0/785440147 >> v1:172.21.15.47:6789/0 conn(0x7fc3ec141b40 legacy=0x7fc3ec13e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 10 monclient: _finish_hunting 0 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 1 monclient: found mon.noname-c 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.152 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.151+0000 7fc3f0eb5640 1 -- 172.21.15.47:0/785440147 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0a8e70 con 0x7fc3ec13ef40 2024-03-30T07:16:23.156 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.156+0000 7fc3f0eb5640 1 --2- 172.21.15.47:0/785440147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec13ef40 0x7fc3ec14be70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b4a70 tx=0x7fc3e402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac609743c1560204 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:23.167 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 1 -- 172.21.15.47:0/785440147 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e402cd00 con 0x7fc3ec13ef40 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: dump: 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:23.168 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 1 -- 172.21.15.47:0/785440147 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e402ce60 con 0x7fc3ec13ef40 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 1 -- 172.21.15.47:0/785440147 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4034660 con 0x7fc3ec13ef40 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3eaffd640 10 monclient: dump: 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:23.169 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3f2918640 5 monclient: authenticate success, global_id 4210 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3f2918640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.164+0000 7fc3f2918640 4 set_mon_vals no callback set 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 10 monclient: shutdown 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 -- 172.21.15.47:0/785440147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec13ef40 msgr2=0x7fc3ec14be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 --2- 172.21.15.47:0/785440147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec13ef40 0x7fc3ec14be70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0b4a70 tx=0x7fc3e402bf50 comp rx=0 tx=0).stop 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 -- 172.21.15.47:0/785440147 shutdown_connections 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 --2- 172.21.15.47:0/785440147 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec13ef40 0x7fc3ec14be70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 --2- 172.21.15.47:0/785440147 >> v2:172.21.15.47:3300/0 conn(0x7fc3ec0b42a0 0x7fc3ec141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.170 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 -- 172.21.15.47:0/785440147 >> 172.21.15.47:0/785440147 conn(0x7fc3ec065d30 msgr2=0x7fc3ec066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 -- 172.21.15.47:0/785440147 shutdown_connections 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.165+0000 7fc3f2918640 1 -- 172.21.15.47:0/785440147 wait complete. 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.166+0000 7fc3f2918640 10 monclient: build_initial_monmap 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.166+0000 7fc3f2918640 10 monclient: monmap: 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.166+0000 7fc3f2918640 1 librados: starting msgr at 2024-03-30T07:16:23.171 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.166+0000 7fc3f2918640 1 librados: starting objecter 2024-03-30T07:16:23.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.167+0000 7fc3f2918640 1 Processor -- start 2024-03-30T07:16:23.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.167+0000 7fc3f2918640 1 -- start start 2024-03-30T07:16:23.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.167+0000 7fc3f2918640 1 librados: setting wanted keys 2024-03-30T07:16:23.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.167+0000 7fc3f2918640 1 librados: calling monclient init 2024-03-30T07:16:23.172 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.167+0000 7fc3f2918640 10 monclient: init 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ec0b42a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc3ec13ef40 0x7fc3ec144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ec13ef40 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.168+0000 7fc3f2918640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec141b40 0x7fc3ec1d9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ec141b40 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:23.177 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1d9770 con 0x7fc3ec141b40 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1d99a0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3eb7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc3ec13ef40 0x7fc3ec144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:60770/0 (socket says 172.21.15.47:60770) 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 1 -- 172.21.15.47:0/3924824045 learned_addr learned my addr 172.21.15.47:0/3924824045 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec1da400 con 0x7fc3ec13ef40 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3f2918640 10 monclient(hunting): authenticate will time out at 1509.284924s 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request con 0x7fc3ec0b42a0 auth_method 0 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:23.178 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3ec13ef40 auth_method 0 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.169+0000 7fc3eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.170+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.170+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.170+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.171+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.171+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.171+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.171+0000 7fc3f0eb5640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec141b40 0x7fc3ec1d9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.171+0000 7fc3f0eb5640 10 monclient(hunting): get_auth_request con 0x7fc3ec141b40 auth_method 0 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:23.179 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.172+0000 7fc3f0eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_done global_id 4213 payload 824 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 -- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc3ec13ef40 msgr2=0x7fc3ec144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc3ec13ef40 0x7fc3ec144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 -- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec141b40 msgr2=0x7fc3ec1d9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec141b40 0x7fc3ec1d9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 monclient: found mon.noname-b 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.173+0000 7fc3ebfff640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4009a70 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.175+0000 7fc3ebfff640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00d830 tx=0x7fc3dc00dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:23.180 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.175+0000 7fc3e8ff9640 10 client.?.objecter ms_handle_connect 0x7fc3ec0b42a0 2024-03-30T07:16:23.181 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.175+0000 7fc3e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00b6c0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:23.183 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: dump: 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: _finish_auth 0 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:23.184 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc00b820 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00bbb0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3e8ff9640 10 monclient: dump: 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:23.185 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3ea7fc640 4 set_mon_vals no callback set 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 5 monclient: authenticate success, global_id 4213 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 10 monclient: _renew_subs 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec145740 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 10 monclient: _renew_subs 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec142900 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.179+0000 7fc3f2918640 1 librados: init done 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.181+0000 7fc3f2918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.181+0000 7fc3f2918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.181+0000 7fc3f2918640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.186 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.181+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec1d9b10 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.460+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fc3dc01e460 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.460+0000 7fc3e8ff9640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 0x7fc3d806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:23.461 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.460+0000 7fc3eb7fe640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 0x7fc3d806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:23.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.461+0000 7fc3eb7fe640 10 monclient: get_auth_request con 0x7fc3d806c550 auth_method 0 2024-03-30T07:16:23.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.461+0000 7fc3e8ff9640 10 client.4213.objecter ms_handle_connect 0x7fc3d806c550 2024-03-30T07:16:23.462 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.461+0000 7fc3eb7fe640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 0x7fc3d806ea10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0abee0 tx=0x7fc3e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc3dc059370 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 10 client.4213.objecter ms_dispatch 0x7fc3ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-30T07:16:23.483 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc3dc0556b0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:23.484 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.482+0000 7fc3e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:23.790 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.790+0000 7fc3f2918640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:23.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.790+0000 7fc3f2918640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:23.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.790+0000 7fc3f2918640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:23.791 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.790+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3ec04e0a0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.818+0000 7fc3e8ff9640 1 -- 172.21.15.47:0/3924824045 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc3dc058cd0 con 0x7fc3ec0b42a0 2024-03-30T07:16:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.818+0000 7fc3e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:23.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.818+0000 7fc3e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:23.819 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:23.820 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":10,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:15:50.083237+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 10 librados: watch_flush enter 2024-03-30T07:16:23.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 10 librados: watch_flush exit 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 10 client.4213.objecter successfully canceled tick 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 msgr2=0x7fc3d806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 0x7fc3d806ea10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0abee0 tx=0x7fc3e0009040 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 10 monclient: shutdown 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 msgr2=0x7fc3ec0aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00d830 tx=0x7fc3dc00dcf0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 shutdown_connections 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc3d806c550 0x7fc3d806ea10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3ec141b40 0x7fc3ec1d9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc3ec13ef40 0x7fc3ec144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 --2- 172.21.15.47:0/3924824045 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc3ec0b42a0 0x7fc3ec0aadc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:23.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 >> 172.21.15.47:0/3924824045 conn(0x7fc3ec065d30 msgr2=0x7fc3ec13ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 shutdown_connections 2024-03-30T07:16:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.821+0000 7fc3f2918640 1 -- 172.21.15.47:0/3924824045 wait complete. 2024-03-30T07:16:23.824 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:23.822+0000 7fc3f2918640 1 librados: shutdown 2024-03-30T07:16:23.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T07:16:25.901 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:16:25.897+0000 5dc2300 -1 osd.7 0 log_to_monitors true 2024-03-30T07:16:25.936 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:16:25.932+0000 5dc2300 -1 osd.6 0 log_to_monitors true 2024-03-30T07:16:26.170 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:16:26.166+0000 5dc2300 -1 osd.5 0 log_to_monitors true 2024-03-30T07:16:26.346 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:16:26.335+0000 5dc2300 -1 osd.4 0 log_to_monitors true 2024-03-30T07:16:26.985 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:16:26.981+0000 5dc2300 -1 osd.3 0 log_to_monitors true 2024-03-30T07:16:27.428 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:16:27.413+0000 5dc2300 -1 osd.1 0 log_to_monitors true 2024-03-30T07:16:27.559 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:27.554+0000 5dc2300 -1 osd.0 0 log_to_monitors true 2024-03-30T07:16:27.724 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:27.714+0000 5dc2300 -1 osd.2 0 log_to_monitors true 2024-03-30T07:16:28.923 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:16:28.918+0000 c496640 -1 osd.6 0 waiting for initial osdmap 2024-03-30T07:16:29.380 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:16:29.366+0000 1025b640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:30.041 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:16:30.036+0000 c496640 -1 osd.3 0 waiting for initial osdmap 2024-03-30T07:16:30.057 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:16:30.050+0000 c496640 -1 osd.5 0 waiting for initial osdmap 2024-03-30T07:16:30.461 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:16:30.447+0000 1025b640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:30.516 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:16:30.501+0000 1025b640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:30.643 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:30.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.735+0000 7efc43592640 10 monclient: get_monmap_and_config 2024-03-30T07:16:30.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.735+0000 7efc43592640 10 monclient: build_initial_monmap 2024-03-30T07:16:30.736 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.735+0000 7efc43592640 10 monclient: monmap: 2024-03-30T07:16:30.736 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:30.736 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:30.736375+0000 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:30.736375+0000 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:30.737 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.736+0000 7efc43592640 1 Processor -- start 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.736+0000 7efc43592640 1 -- start start 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.736+0000 7efc43592640 10 monclient: init 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efc3c0b43c0 0x7efc3c0a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): picked mon.noname-a con 0x7efc3c0b43c0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc3c13e790 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc3c141670 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c13f2d0 con 0x7efc3c141670 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3c1421f0 con 0x7efc3c13e790 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:30.739 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3c0b47a0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:30.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc43592640 10 monclient(hunting): authenticate will time out at 1516.852269s 2024-03-30T07:16:30.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc40b2d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7efc3c13e790 0x7efc3c13ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:60652/0 (socket says 172.21.15.47:60652) 2024-03-30T07:16:30.740 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.737+0000 7efc40b2d640 1 -- 172.21.15.47:0/1601271924 learned_addr learned my addr 172.21.15.47:0/1601271924 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:30.793 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.793+0000 7efc4132e640 1 --2- 172.21.15.47:0/1601271924 >> v2:172.21.15.47:3300/0 conn(0x7efc3c0b43c0 0x7efc3c0a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:30.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.793+0000 7efc4132e640 10 monclient(hunting): get_auth_request con 0x7efc3c0b43c0 auth_method 0 2024-03-30T07:16:30.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.793+0000 7efc4132e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:30.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.793+0000 7efc4132e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:30.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.793+0000 7efc4132e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:30.794 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.794+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:30.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.794+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:30.795 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.794+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 1 -- 172.21.15.47:0/1601271924 >> v1:172.21.15.52:6789/0 conn(0x7efc3c13e790 legacy=0x7efc3c13ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 1 -- 172.21.15.47:0/1601271924 >> v1:172.21.15.47:6789/0 conn(0x7efc3c141670 legacy=0x7efc3c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 10 monclient: _finish_hunting 0 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 1 monclient: found mon.noname-a 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:30.798 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.797+0000 7efc4132e640 1 -- 172.21.15.47:0/1601271924 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c0a95a0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.800 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.800+0000 7efc4132e640 1 --2- 172.21.15.47:0/1601271924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0a8990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc380097a0 tx=0x7efc3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c079f6d40e1351 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 1 -- 172.21.15.47:0/1601271924 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38031830 con 0x7efc3c0b43c0 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: dump: 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:30.806 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: _finish_auth 0 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 1 -- 172.21.15.47:0/1601271924 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc38031990 con 0x7efc3c0b43c0 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc43592640 5 monclient: authenticate success, global_id 4227 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc43592640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc43592640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 1 -- 172.21.15.47:0/1601271924 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38031d20 con 0x7efc3c0b43c0 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc43592640 4 set_mon_vals no callback set 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:30.807 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.805+0000 7efc33fff640 10 monclient: dump: 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:30.808 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 10 monclient: shutdown 2024-03-30T07:16:30.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:30.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 -- 172.21.15.47:0/1601271924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 msgr2=0x7efc3c0a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:30.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 --2- 172.21.15.47:0/1601271924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0a8990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efc380097a0 tx=0x7efc3802b8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:30.811 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 -- 172.21.15.47:0/1601271924 shutdown_connections 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 --2- 172.21.15.47:0/1601271924 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0a8990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 -- 172.21.15.47:0/1601271924 >> 172.21.15.47:0/1601271924 conn(0x7efc3c065f70 msgr2=0x7efc3c066380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 -- 172.21.15.47:0/1601271924 shutdown_connections 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.806+0000 7efc43592640 1 -- 172.21.15.47:0/1601271924 wait complete. 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 10 monclient: build_initial_monmap 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 10 monclient: monmap: 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:30.812 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 librados: starting msgr at 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 librados: starting objecter 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 Processor -- start 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 -- start start 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 librados: setting wanted keys 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 1 librados: calling monclient init 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.807+0000 7efc43592640 10 monclient: init 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:30.813 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): picked mon.noname-a con 0x7efc3c0b43c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc3c13e790 0x7efc3c1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): picked mon.noname-c con 0x7efc3c13e790 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc3c141670 0x7efc3c138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): picked mon.noname-b con 0x7efc3c141670 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c138500 con 0x7efc3c0b43c0 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c1448f0 con 0x7efc3c141670 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c144bd0 con 0x7efc3c13e790 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc43592640 10 monclient(hunting): authenticate will time out at 1516.923316s 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.808+0000 7efc4132e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:30.814 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33370/0 (socket says 172.21.15.47:33370) 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 1 -- 172.21.15.47:0/2359108646 learned_addr learned my addr 172.21.15.47:0/2359108646 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): get_auth_request con 0x7efc3c0b43c0 auth_method 0 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:30.815 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.809+0000 7efc4132e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:30.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-03-30T07:16:30.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 -- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc3c13e790 msgr2=0x7efc3c1450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:30.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc3c13e790 0x7efc3c1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:30.816 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 -- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc3c141670 msgr2=0x7efc3c138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc3c141670 0x7efc3c138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 10 monclient: _finish_hunting 0 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 monclient: found mon.noname-a 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.812+0000 7efc4132e640 1 -- 172.21.15.47:0/2359108646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc38009400 con 0x7efc3c0b43c0 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.815+0000 7efc4132e640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc3802bee0 tx=0x7efc380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.815+0000 7efc31ffb640 10 client.?.objecter ms_handle_connect 0x7efc3c0b43c0 2024-03-30T07:16:30.817 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.815+0000 7efc31ffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:30.818 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3803ba70 con 0x7efc3c0b43c0 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 10 monclient: dump: 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:30.819 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc43592640 5 monclient: authenticate success, global_id 4233 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc43592640 10 monclient: _renew_subs 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc43592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.817+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c145630 con 0x7efc3c0b43c0 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.818+0000 7efc43592640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.818+0000 7efc43592640 10 monclient: _renew_subs 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.818+0000 7efc43592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:30.820 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.818+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c1445f0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.818+0000 7efc43592640 1 librados: init done 2024-03-30T07:16:30.821 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc43592640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc43592640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc43592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c144df0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3803bbd0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380453f0 con 0x7efc3c0b43c0 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc31ffb640 10 monclient: dump: 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.822 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:30.823 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:30.819+0000 7efc337fe640 4 set_mon_vals no callback set 2024-03-30T07:16:31.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.094+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7efc3804d3f0 con 0x7efc3c0b43c0 2024-03-30T07:16:31.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.094+0000 7efc31ffb640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 0x7efc2006e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:31.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.094+0000 7efc40b2d640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 0x7efc2006e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:31.095 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.094+0000 7efc40b2d640 10 monclient: get_auth_request con 0x7efc2006c500 auth_method 0 2024-03-30T07:16:31.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.095+0000 7efc40b2d640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 0x7efc2006e9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0b7b00 tx=0x7efc2c007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:31.096 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.095+0000 7efc31ffb640 10 client.4233.objecter ms_handle_connect 0x7efc2006c500 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2183+0+0 (secure 0 0 0) 0x7efc3804b2c0 con 0x7efc3c0b43c0 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 10 client.4233.objecter ms_dispatch 0x7efc3c000b90 osd_map(13..13 src has 1..13) v4 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 3 client.4233.objecter handle_osd_map got epochs [13,13] > 0 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 3 client.4233.objecter handle_osd_map decoding full epoch 13 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 20 client.4233.objecter dump_active .. 0 homeless 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efc380766b0 con 0x7efc3c0b43c0 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:31.111 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.110+0000 7efc31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:31.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.306+0000 7efc43592640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:31.307 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.306+0000 7efc43592640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:31.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.306+0000 7efc43592640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:31.308 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.306+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc3c138640 con 0x7efc3c0b43c0 2024-03-30T07:16:31.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.349+0000 7efc31ffb640 1 -- 172.21.15.47:0/2359108646 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8322 (secure 0 0 0) 0x7efc3c138640 con 0x7efc3c0b43c0 2024-03-30T07:16:31.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.349+0000 7efc31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:31.350 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.349+0000 7efc31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:31.350 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:31.350 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":14,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:16:30.709385+0000","last_up_change":"2024-03-30T07:16:30.709385+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":35314},{"type":"v1","addr":"172.21.15.52:6829","nonce":35314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6829/35314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":35315},{"type":"v1","addr":"172.21.15.52:6807","nonce":35315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6807/35315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.351+0000 7efc43592640 10 librados: watch_flush enter 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.351+0000 7efc43592640 10 librados: watch_flush exit 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 20 client.4233.objecter shutdown clearing up homeless session... 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 10 client.4233.objecter successfully canceled tick 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 msgr2=0x7efc2006e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 0x7efc2006e9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0b7b00 tx=0x7efc2c007a40 comp rx=0 tx=0).stop 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 10 monclient: shutdown 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 msgr2=0x7efc3c0b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:31.353 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efc3802bee0 tx=0x7efc380046c0 comp rx=0 tx=0).stop 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 shutdown_connections 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efc2006c500 0x7efc2006e9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efc3c141670 0x7efc3c138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc3c13e790 0x7efc3c1450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 --2- 172.21.15.47:0/2359108646 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc3c0b43c0 0x7efc3c0b7460 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 >> 172.21.15.47:0/2359108646 conn(0x7efc3c065f70 msgr2=0x7efc3c140600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 shutdown_connections 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 -- 172.21.15.47:0/2359108646 wait complete. 2024-03-30T07:16:31.354 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:31.352+0000 7efc43592640 1 librados: shutdown 2024-03-30T07:16:31.366 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-03-30T07:16:31.682 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:16:31.678+0000 ac7d640 -1 osd.1 0 waiting for initial osdmap 2024-03-30T07:16:32.086 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:16:32.073+0000 1025b640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:37.415 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:16:36.714+0000 5dc2300 -1 bdev(0x1916ce20 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-03-30T07:16:37.784 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:16:37.767+0000 5dc2300 -1 bdev(0x18e945c0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-03-30T07:16:38.172 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:38.263 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.262+0000 7fc52fd11640 10 monclient: get_monmap_and_config 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.262+0000 7fc52fd11640 10 monclient: build_initial_monmap 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.262+0000 7fc52fd11640 10 monclient: monmap: 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:38.263841+0000 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:38.263841+0000 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:38.264 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.262+0000 7fc52fd11640 1 Processor -- start 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 -- start start 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient: init 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc5280b42a0 0x7fc528141600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-c con 0x7fc5280b42a0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 0x7fc52813e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.267 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-b con 0x7fc528141b40 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc52813ed70 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52813f200 con 0x7fc52813ed70 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc528141fa0 con 0x7fc5280b42a0 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b4680 con 0x7fc528141b40 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.263+0000 7fc52fd11640 10 monclient(hunting): authenticate will time out at 1524.379811s 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52d2ac640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 0x7fc52813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52daad640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc5280b42a0 0x7fc528141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52e2ae640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc52813ed70 0x7fc528144b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43844/0 (socket says 172.21.15.47:43844) 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52e2ae640 1 -- 172.21.15.47:0/876846209 learned_addr learned my addr 172.21.15.47:0/876846209 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52d2ac640 10 monclient(hunting): get_auth_request con 0x7fc528141b40 auth_method 0 2024-03-30T07:16:38.268 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52d2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52d2ac640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52d2ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52daad640 10 monclient(hunting): get_auth_request con 0x7fc5280b42a0 auth_method 0 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52daad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52daad640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.264+0000 7fc52daad640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:38.269 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.265+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 1 -- 172.21.15.47:0/876846209 >> v2:172.21.15.52:3300/0 conn(0x7fc5280b42a0 msgr2=0x7fc528141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 1 --2- 172.21.15.47:0/876846209 >> v2:172.21.15.52:3300/0 conn(0x7fc5280b42a0 0x7fc528141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 1 -- 172.21.15.47:0/876846209 >> v1:172.21.15.47:6789/0 conn(0x7fc52813ed70 legacy=0x7fc528144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 10 monclient: _finish_hunting 0 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 1 monclient: found mon.noname-b 2024-03-30T07:16:38.270 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:38.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.266+0000 7fc52d2ac640 1 -- 172.21.15.47:0/876846209 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280aa6a0 con 0x7fc528141b40 2024-03-30T07:16:38.271 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.268+0000 7fc52d2ac640 1 --2- 172.21.15.47:0/876846209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 0x7fc52813e830 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30606d1553df1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 1 -- 172.21.15.47:0/876846209 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c031830 con 0x7fc528141b40 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: dump: 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.274 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: _finish_auth 0 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: _check_auth_tickets 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 1 -- 172.21.15.47:0/876846209 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c031990 con 0x7fc528141b40 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 1 -- 172.21.15.47:0/876846209 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c031d20 con 0x7fc528141b40 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52caab640 10 monclient: dump: 2024-03-30T07:16:38.275 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52fd11640 5 monclient: authenticate success, global_id 4193 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52fd11640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:38.276 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.272+0000 7fc52fd11640 4 set_mon_vals no callback set 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 10 monclient: shutdown 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 -- 172.21.15.47:0/876846209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 msgr2=0x7fc52813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 --2- 172.21.15.47:0/876846209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 0x7fc52813e830 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c02b8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 -- 172.21.15.47:0/876846209 shutdown_connections 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 --2- 172.21.15.47:0/876846209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc528141b40 0x7fc52813e830 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 --2- 172.21.15.47:0/876846209 >> v2:172.21.15.52:3300/0 conn(0x7fc5280b42a0 0x7fc528141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 -- 172.21.15.47:0/876846209 >> 172.21.15.47:0/876846209 conn(0x7fc528065d30 msgr2=0x7fc528066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:38.285 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 -- 172.21.15.47:0/876846209 shutdown_connections 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.273+0000 7fc52fd11640 1 -- 172.21.15.47:0/876846209 wait complete. 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 10 monclient: build_initial_monmap 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 10 monclient: monmap: 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.286 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 librados: starting msgr at 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 librados: starting objecter 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 Processor -- start 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 -- start start 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 librados: setting wanted keys 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 1 librados: calling monclient init 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.274+0000 7fc52fd11640 10 monclient: init 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-c con 0x7fc5280b42a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:38.287 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc52813ed70 0x7fc528142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-a con 0x7fc52813ed70 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.275+0000 7fc52fd11640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc528141b40 0x7fc5281de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): picked mon.noname-b con 0x7fc528141b40 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b5150 con 0x7fc52813ed70 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.288 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b64d0 con 0x7fc528141b40 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280b67b0 con 0x7fc5280b42a0 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52fd11640 10 monclient(hunting): authenticate will time out at 1524.392652s 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51290/0 (socket says 172.21.15.47:51290) 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 1 -- 172.21.15.47:0/4247080972 learned_addr learned my addr 172.21.15.47:0/4247080972 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 10 monclient(hunting): get_auth_request con 0x7fc5280b42a0 auth_method 0 2024-03-30T07:16:38.289 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52daad640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.276+0000 7fc52d2ac640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc52813ed70 0x7fc528142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): get_auth_request con 0x7fc52813ed70 auth_method 0 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52daad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52e2ae640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc528141b40 0x7fc5281de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52e2ae640 10 monclient(hunting): get_auth_request con 0x7fc528141b40 auth_method 0 2024-03-30T07:16:38.290 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52e2ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52e2ae640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52e2ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.277+0000 7fc52d2ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.278+0000 7fc52e2ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.278+0000 7fc52e2ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.278+0000 7fc52e2ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 -- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc528141b40 msgr2=0x7fc5281de6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc528141b40 0x7fc5281de6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 -- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc52813ed70 msgr2=0x7fc528142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc52813ed70 0x7fc528142900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:38.291 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 10 monclient: _finish_hunting 0 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 monclient: found mon.noname-c 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.280+0000 7fc52daad640 1 -- 172.21.15.47:0/4247080972 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009400 con 0x7fc5280b42a0 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.282+0000 7fc52daad640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc52000e870 tx=0x7fc52000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.282+0000 7fc5267fc640 10 client.?.objecter ms_handle_connect 0x7fc5280b42a0 2024-03-30T07:16:38.292 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.282+0000 7fc5267fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5200094f0 con 0x7fc5280b42a0 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 10 monclient: dump: 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.294 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52000c940 con 0x7fc5280b42a0 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc52fd11640 5 monclient: authenticate success, global_id 4196 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc52fd11640 10 monclient: _renew_subs 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc52fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.286+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5281445b0 con 0x7fc5280b42a0 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc5267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52000ccf0 con 0x7fc5280b42a0 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc5267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:38.295 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc5267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc5267fc640 10 monclient: dump: 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc52fd11640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc52fd11640 10 monclient: _renew_subs 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc52fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.296 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc527fff640 4 set_mon_vals no callback set 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc528145180 con 0x7fc5280b42a0 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.287+0000 7fc52fd11640 1 librados: init done 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.288+0000 7fc52fd11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.288+0000 7fc52fd11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.288+0000 7fc52fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.297 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.288+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5280b52c0 con 0x7fc5280b42a0 2024-03-30T07:16:38.504 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:16:38.064+0000 5dc2300 -1 bdev(0x18e945c0 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-03-30T07:16:38.566 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:16:38.559+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2024-03-30T07:16:38.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.645+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7fc520013470 con 0x7fc5280b42a0 2024-03-30T07:16:38.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.645+0000 7fc5267fc640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 0x7fc51406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:38.647 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.646+0000 7fc52d2ac640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 0x7fc51406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:38.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.646+0000 7fc52d2ac640 10 monclient: get_auth_request con 0x7fc51406c500 auth_method 0 2024-03-30T07:16:38.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.646+0000 7fc52d2ac640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 0x7fc51406e9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c03c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:38.648 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.646+0000 7fc5267fc640 10 client.4196.objecter ms_handle_connect 0x7fc51406c500 2024-03-30T07:16:38.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 3737+0+0 (secure 0 0 0) 0x7fc520058930 con 0x7fc5280b42a0 2024-03-30T07:16:38.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 10 client.4196.objecter ms_dispatch 0x7fc528000b90 osd_map(18..18 src has 1..18) v4 2024-03-30T07:16:38.665 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 3 client.4196.objecter handle_osd_map got epochs [18,18] > 0 2024-03-30T07:16:38.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 3 client.4196.objecter handle_osd_map decoding full epoch 18 2024-03-30T07:16:38.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-30T07:16:38.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc520054c70 con 0x7fc5280b42a0 2024-03-30T07:16:38.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:38.666 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.663+0000 7fc5267fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:38.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.902+0000 7fc52fd11640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:38.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.902+0000 7fc52fd11640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:38.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.902+0000 7fc52fd11640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:38.904 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:38.902+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5280b52c0 con 0x7fc5280b42a0 2024-03-30T07:16:39.014 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:16:38.995+0000 14263640 -1 osd.7 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:39.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.035+0000 7fc5267fc640 1 -- 172.21.15.47:0/4247080972 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+10587 (secure 0 0 0) 0x7fc5280b52c0 con 0x7fc5280b42a0 2024-03-30T07:16:39.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.035+0000 7fc5267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:39.037 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.035+0000 7fc5267fc640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:39.037 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:39.038 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":18,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:16:37.623141+0000","last_up_change":"2024-03-30T07:16:33.439642+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T07:16:32.182687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":35204},{"type":"v1","addr":"172.21.15.47:6809","nonce":35204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":35204},{"type":"v1","addr":"172.21.15.47:6813","nonce":35204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":35204},{"type":"v1","addr":"172.21.15.47:6820","nonce":35204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":35204},{"type":"v1","addr":"172.21.15.47:6816","nonce":35204}]},"public_addr":"172.21.15.47:6809/35204","cluster_addr":"172.21.15.47:6813/35204","heartbeat_back_addr":"172.21.15.47:6820/35204","heartbeat_front_addr":"172.21.15.47:6816/35204","state":["exists","up"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":35314},{"type":"v1","addr":"172.21.15.52:6829","nonce":35314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6829/35314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":35315},{"type":"v1","addr":"172.21.15.52:6807","nonce":35315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6807/35315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.651813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.080129+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.240245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.041599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 10 librados: watch_flush enter 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 10 librados: watch_flush exit 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 10 client.4196.objecter successfully canceled tick 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 msgr2=0x7fc51406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 0x7fc51406e9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c03c040 comp rx=0 tx=0).stop 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 10 monclient: shutdown 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 msgr2=0x7fc5281de100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc52000e870 tx=0x7fc52000ed30 comp rx=0 tx=0).stop 2024-03-30T07:16:39.042 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 shutdown_connections 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc51406c500 0x7fc51406e9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc528141b40 0x7fc5281de6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc52813ed70 0x7fc528142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 --2- 172.21.15.47:0/4247080972 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5280b42a0 0x7fc5281de100 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 >> 172.21.15.47:0/4247080972 conn(0x7fc528065d30 msgr2=0x7fc52813ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.038+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 shutdown_connections 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.039+0000 7fc52fd11640 1 -- 172.21.15.47:0/4247080972 wait complete. 2024-03-30T07:16:39.043 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:39.039+0000 7fc52fd11640 1 librados: shutdown 2024-03-30T07:16:39.056 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-03-30T07:16:39.653 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:16:39.624+0000 c496640 -1 osd.4 0 waiting for initial osdmap 2024-03-30T07:16:40.122 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:16:40.108+0000 14263640 -1 osd.4 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:40.263 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:39.954+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 3 2024-03-30T07:16:40.265 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:39.989+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.278 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.265+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-03-30T07:16:40.323 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.277+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.323 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.289+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.477 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.322+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.628 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:39.980+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.691 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:40.627+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.714 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:40.678+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.743 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:40.712+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.743 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:40.729+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.835 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.476+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.835 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.696+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.860 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:40.742+0000 5dc2300 -1 bdev(0x18e91840 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-03-30T07:16:40.872 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.854+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.899 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.871+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.967 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.897+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:40.968 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:40.924+0000 5dc2300 -1 bdev(0x1651ab60 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-30T07:16:41.648 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:09.460 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:41.730 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:09.542 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:41.761 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:09.572 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:41.766 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:09.545 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:41.816 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:09.627 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:41.829 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:09.607 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:42.031 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:42.025+0000 ac7d640 -1 osd.0 0 waiting for initial osdmap 2024-03-30T07:16:42.066 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:09.878 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:42.095 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:09.873 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:42.096 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:09.908 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:16:42.187 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:42.180+0000 c496640 -1 osd.2 0 waiting for initial osdmap 2024-03-30T07:16:42.598 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:16:42.582+0000 14263640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:42.797 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:16:42.781+0000 14263640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:16:45.864 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:45.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.952+0000 7fced04df640 10 monclient: get_monmap_and_config 2024-03-30T07:16:45.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.952+0000 7fced04df640 10 monclient: build_initial_monmap 2024-03-30T07:16:45.954 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.952+0000 7fced04df640 10 monclient: monmap: 2024-03-30T07:16:45.954 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:45.954 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:16:45.954480+0000 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:16:45.954480+0000 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:45.955 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.953+0000 7fced04df640 1 Processor -- start 2024-03-30T07:16:45.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.953+0000 7fced04df640 1 -- start start 2024-03-30T07:16:45.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.953+0000 7fced04df640 10 monclient: init 2024-03-30T07:16:45.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:45.957 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcec80b4240 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcec80a89f0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fcec8141c70 0x7fcec814ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): picked mon.noname-c con 0x7fcec8141c70 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec80aa630 con 0x7fcec80a89f0 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec80a8fa0 con 0x7fcec80b4240 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec80b4620 con 0x7fcec8141c70 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fced04df640 10 monclient(hunting): authenticate will time out at 1532.070241s 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fcecda7a640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcec80a89f0 0x7fcec8141560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43860/0 (socket says 172.21.15.47:43860) 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fcecda7a640 1 -- 172.21.15.47:0/2825460772 learned_addr learned my addr 172.21.15.47:0/2825460772 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.954+0000 7fcecea7c640 1 --2- 172.21.15.47:0/2825460772 >> v2:172.21.15.52:3300/0 conn(0x7fcec8141c70 0x7fcec814ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:45.958 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.955+0000 7fcecea7c640 10 monclient(hunting): get_auth_request con 0x7fcec8141c70 auth_method 0 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.955+0000 7fcecea7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.955+0000 7fcecea7c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.955+0000 7fcecea7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.956+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.956+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:45.959 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.956+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 1 -- 172.21.15.47:0/2825460772 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079547289 0 0) 0x7fcec80aa630 con 0x7fcec80a89f0 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 10 monclient(hunting): _negotiate 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 10 monclient(hunting): my global_id is 4248 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.957+0000 7fcecd279640 1 -- 172.21.15.47:0/2825460772 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb4003520 con 0x7fcec80a89f0 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-03-30T07:16:45.960 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 1 -- 172.21.15.47:0/2825460772 >> v1:172.21.15.52:6789/0 conn(0x7fcec80b4240 legacy=0x7fcec813e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 1 -- 172.21.15.47:0/2825460772 >> v1:172.21.15.47:6789/0 conn(0x7fcec80a89f0 legacy=0x7fcec8141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 1 monclient: found mon.noname-c 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:45.961 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.958+0000 7fcecea7c640 1 -- 172.21.15.47:0/2825460772 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec80b3980 con 0x7fcec8141c70 2024-03-30T07:16:45.965 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.963+0000 7fcecea7c640 1 --2- 172.21.15.47:0/2825460772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec8141c70 0x7fcec814ce20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009d90 tx=0x7fcec002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b0946628fb6313 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 1 -- 172.21.15.47:0/2825460772 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec002cc60 con 0x7fcec8141c70 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: dump: 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:45.969 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: _finish_auth 0 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: _check_auth_tickets 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 1 -- 172.21.15.47:0/2825460772 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec002cdc0 con 0x7fcec8141c70 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 1 -- 172.21.15.47:0/2825460772 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0034660 con 0x7fcec8141c70 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fced04df640 5 monclient: authenticate success, global_id 4249 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fced04df640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fced04df640 4 set_mon_vals no callback set 2024-03-30T07:16:45.970 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.967+0000 7fcecd279640 10 monclient: dump: 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:45.971 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 10 monclient: shutdown 2024-03-30T07:16:45.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:45.974 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 -- 172.21.15.47:0/2825460772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec8141c70 msgr2=0x7fcec814ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 --2- 172.21.15.47:0/2825460772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec8141c70 0x7fcec814ce20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009d90 tx=0x7fcec002beb0 comp rx=0 tx=0).stop 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 -- 172.21.15.47:0/2825460772 shutdown_connections 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 --2- 172.21.15.47:0/2825460772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec8141c70 0x7fcec814ce20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 -- 172.21.15.47:0/2825460772 >> 172.21.15.47:0/2825460772 conn(0x7fcec8065d30 msgr2=0x7fcec8066140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 -- 172.21.15.47:0/2825460772 shutdown_connections 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 1 -- 172.21.15.47:0/2825460772 wait complete. 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 10 monclient: build_initial_monmap 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.968+0000 7fced04df640 10 monclient: monmap: 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-03-30T07:16:45.975 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 librados: starting msgr at 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 librados: starting objecter 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 Processor -- start 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 -- start start 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 librados: setting wanted keys 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 librados: calling monclient init 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 10 monclient: init 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:45.976 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 0x7fcec81d9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 10 monclient(hunting): picked mon.noname-b con 0x7fcec80a89f0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcec80b4240 0x7fcec81373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.969+0000 7fced04df640 10 monclient(hunting): picked mon.noname-c con 0x7fcec80b4240 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): picked mon.noname-a con 0x7fcec8141c70 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec80ae7e0 con 0x7fcec8141c70 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec8137e50 con 0x7fcec80a89f0 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec80ac530 con 0x7fcec80b4240 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fced04df640 10 monclient(hunting): authenticate will time out at 1532.086162s 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:45.977 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecda7a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcec80b4240 0x7fcec81373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcece27b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 0x7fcec81d9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:56986/0 (socket says 172.21.15.47:56986) 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 1 -- 172.21.15.47:0/1524007629 learned_addr learned my addr 172.21.15.47:0/1524007629 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 10 monclient(hunting): get_auth_request con 0x7fcec8141c70 auth_method 0 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecea7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcece27b640 10 monclient(hunting): get_auth_request con 0x7fcec80a89f0 auth_method 0 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcece27b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcece27b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcece27b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecda7a640 10 monclient(hunting): get_auth_request con 0x7fcec80b4240 auth_method 0 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecda7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecda7a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:45.978 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.970+0000 7fcecda7a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecea7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcece27b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcece27b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcece27b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecda7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecda7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:45.979 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.971+0000 7fcecda7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 -- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcec80b4240 msgr2=0x7fcec81373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcec80b4240 0x7fcec81373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 -- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 msgr2=0x7fcec81d9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 0x7fcec81d9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 monclient: found mon.noname-a 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcecea7c640 1 -- 172.21.15.47:0/1524007629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec0009a70 con 0x7fcec8141c70 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.974+0000 7fcece27b640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 0x7fcec81d9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.976+0000 7fcecea7c640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c4c0 tx=0x7fcec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:45.981 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.976+0000 7fcebeffd640 10 client.?.objecter ms_handle_connect 0x7fcec8141c70 2024-03-30T07:16:45.982 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.976+0000 7fcebeffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec003e610 con 0x7fcec8141c70 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: dump: 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.983 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec003e770 con 0x7fcec8141c70 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec003eb20 con 0x7fcec8141c70 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 5 monclient: authenticate success, global_id 4251 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 10 monclient: _renew_subs 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec80a9640 con 0x7fcec8141c70 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebffff640 4 set_mon_vals no callback set 2024-03-30T07:16:45.984 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fcebeffd640 10 monclient: dump: 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr: 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 10 monclient: _renew_subs 2024-03-30T07:16:45.985 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec81da3a0 con 0x7fcec8141c70 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.980+0000 7fced04df640 1 librados: init done 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.981+0000 7fced04df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.981+0000 7fced04df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.981+0000 7fced04df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:45.986 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.981+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec80ad250 con 0x7fcec8141c70 2024-03-30T07:16:46.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.999+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcec003b050 con 0x7fcec8141c70 2024-03-30T07:16:46.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:45.999+0000 7fcebeffd640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 0x7fceac06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.000+0000 7fcece27b640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 0x7fceac06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.001 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.000+0000 7fcece27b640 10 monclient: get_auth_request con 0x7fceac06c5e0 auth_method 0 2024-03-30T07:16:46.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.000+0000 7fcece27b640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 0x7fceac06eaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001b70 tx=0x7fceb8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:46.002 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.000+0000 7fcebeffd640 10 client.4251.objecter ms_handle_connect 0x7fceac06c5e0 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcec007aee0 con 0x7fcec8141c70 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 10 client.4251.objecter ms_dispatch 0x7fcec8000b90 osd_map(23..23 src has 1..23) v4 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 3 client.4251.objecter handle_osd_map got epochs [23,23] > 0 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 3 client.4251.objecter handle_osd_map decoding full epoch 23 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fcec0077220 con 0x7fcec8141c70 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:46.273 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.271+0000 7fcebeffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:46.455 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.453+0000 7fced04df640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:46.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.453+0000 7fced04df640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:46.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.453+0000 7fced04df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:46.456 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.453+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcec804e0a0 con 0x7fcec8141c70 2024-03-30T07:16:46.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.546+0000 7fcebeffd640 1 -- 172.21.15.47:0/1524007629 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+12335 (secure 0 0 0) 0x7fcec004e090 con 0x7fcec8141c70 2024-03-30T07:16:46.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.546+0000 7fcebeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:46.548 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.546+0000 7fcebeffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:46.549 INFO:teuthology.misc.health.smithi047.stdout: 2024-03-30T07:16:46.549 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":23,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:16:45.761984+0000","last_up_change":"2024-03-30T07:16:44.160131+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T07:16:32.182687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":35205},{"type":"v1","addr":"172.21.15.47:6801","nonce":35205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":35205},{"type":"v1","addr":"172.21.15.47:6803","nonce":35205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":35205},{"type":"v1","addr":"172.21.15.47:6807","nonce":35205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":35205},{"type":"v1","addr":"172.21.15.47:6805","nonce":35205}]},"public_addr":"172.21.15.47:6801/35205","cluster_addr":"172.21.15.47:6803/35205","heartbeat_back_addr":"172.21.15.47:6807/35205","heartbeat_front_addr":"172.21.15.47:6805/35205","state":["exists","up"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":35204},{"type":"v1","addr":"172.21.15.47:6809","nonce":35204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":35204},{"type":"v1","addr":"172.21.15.47:6813","nonce":35204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":35204},{"type":"v1","addr":"172.21.15.47:6820","nonce":35204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":35204},{"type":"v1","addr":"172.21.15.47:6816","nonce":35204}]},"public_addr":"172.21.15.47:6809/35204","cluster_addr":"172.21.15.47:6813/35204","heartbeat_back_addr":"172.21.15.47:6820/35204","heartbeat_front_addr":"172.21.15.47:6816/35204","state":["exists","up"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":35207},{"type":"v1","addr":"172.21.15.47:6825","nonce":35207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":35207},{"type":"v1","addr":"172.21.15.47:6827","nonce":35207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":35207},{"type":"v1","addr":"172.21.15.47:6831","nonce":35207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":35207},{"type":"v1","addr":"172.21.15.47:6829","nonce":35207}]},"public_addr":"172.21.15.47:6825/35207","cluster_addr":"172.21.15.47:6827/35207","heartbeat_back_addr":"172.21.15.47:6831/35207","heartbeat_front_addr":"172.21.15.47:6829/35207","state":["exists","up"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":35313},{"type":"v1","addr":"172.21.15.52:6819","nonce":35313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":35313},{"type":"v1","addr":"172.21.15.52:6821","nonce":35313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":35313},{"type":"v1","addr":"172.21.15.52:6825","nonce":35313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":35313},{"type":"v1","addr":"172.21.15.52:6823","nonce":35313}]},"public_addr":"172.21.15.52:6819/35313","cluster_addr":"172.21.15.52:6821/35313","heartbeat_back_addr":"172.21.15.52:6825/35313","heartbeat_front_addr":"172.21.15.52:6823/35313","state":["exists","up"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":35314},{"type":"v1","addr":"172.21.15.52:6829","nonce":35314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6829/35314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":35315},{"type":"v1","addr":"172.21.15.52:6807","nonce":35315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6807/35315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":35316},{"type":"v1","addr":"172.21.15.52:6806","nonce":35316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":35316},{"type":"v1","addr":"172.21.15.52:6812","nonce":35316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":35316},{"type":"v1","addr":"172.21.15.52:6817","nonce":35316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":35316},{"type":"v1","addr":"172.21.15.52:6815","nonce":35316}]},"public_addr":"172.21.15.52:6806/35316","cluster_addr":"172.21.15.52:6812/35316","heartbeat_back_addr":"172.21.15.52:6817/35316","heartbeat_front_addr":"172.21.15.52:6815/35316","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.613520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.651813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.749978+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.080129+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.393136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.240245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.041599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:27.954523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 10 librados: watch_flush enter 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 10 librados: watch_flush exit 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 10 client.4251.objecter successfully canceled tick 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 msgr2=0x7fceac06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 0x7fceac06eaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001b70 tx=0x7fceb8009040 comp rx=0 tx=0).stop 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 10 monclient: shutdown 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 msgr2=0x7fcec8137910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c4c0 tx=0x7fcec0004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 shutdown_connections 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fceac06c5e0 0x7fceac06eaa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.551 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcec8141c70 0x7fcec8137910 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcec80b4240 0x7fcec81373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 --2- 172.21.15.47:0/1524007629 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fcec80a89f0 0x7fcec81d9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 >> 172.21.15.47:0/1524007629 conn(0x7fcec8065d30 msgr2=0x7fcec80b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 shutdown_connections 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.548+0000 7fced04df640 1 -- 172.21.15.47:0/1524007629 wait complete. 2024-03-30T07:16:46.552 INFO:teuthology.misc.health.smithi047.stderr:2024-03-30T07:16:46.549+0000 7fced04df640 1 librados: shutdown 2024-03-30T07:16:46.565 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-30T07:16:46.565 INFO:tasks.ceph:Creating RBD pool 2024-03-30T07:16:46.565 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-30T07:16:46.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 10 monclient: get_monmap_and_config 2024-03-30T07:16:46.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 10 monclient: build_initial_monmap 2024-03-30T07:16:46.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 10 monclient: monmap: 2024-03-30T07:16:46.694 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:46.693802+0000 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:46.693802+0000 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 1 Processor -- start 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 1 -- start start 2024-03-30T07:16:46.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.692+0000 7fc61954e640 10 monclient: init 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc6140b0120 0x7fc6140a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-a con 0x7fc6140b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc614144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc6140b0d20 0x7fc61414be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-c con 0x7fc6140b0d20 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6140a8bd0 con 0x7fc614144ab0 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140aa4f0 con 0x7fc6140b0120 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:46.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc614144f10 con 0x7fc6140b0d20 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc61954e640 10 monclient(hunting): authenticate will time out at 1532.809448s 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.693+0000 7fc612ffd640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc6140b0120 0x7fc6140a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc6140b0120 0x7fc6140a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50842/0 (socket says 172.21.15.47:50842) 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 1 -- 172.21.15.47:0/987320507 learned_addr learned my addr 172.21.15.47:0/987320507 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): get_auth_request con 0x7fc6140b0120 auth_method 0 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc6137fe640 1 --2- 172.21.15.47:0/987320507 >> v2:172.21.15.52:3300/0 conn(0x7fc6140b0d20 0x7fc61414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc6137fe640 10 monclient(hunting): get_auth_request con 0x7fc6140b0d20 auth_method 0 2024-03-30T07:16:46.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc6137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc6137fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.694+0000 7fc6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.695+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.695+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.695+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 1 -- 172.21.15.47:0/987320507 >> v2:172.21.15.52:3300/0 conn(0x7fc6140b0d20 msgr2=0x7fc61414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 1 --2- 172.21.15.47:0/987320507 >> v2:172.21.15.52:3300/0 conn(0x7fc6140b0d20 0x7fc61414be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 1 -- 172.21.15.47:0/987320507 >> v1:172.21.15.52:6789/0 conn(0x7fc614144ab0 legacy=0x7fc6140712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 1 monclient: found mon.noname-a 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:46.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.696+0000 7fc612ffd640 1 -- 172.21.15.47:0/987320507 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6140b3780 con 0x7fc6140b0120 2024-03-30T07:16:46.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.698+0000 7fc612ffd640 1 --2- 172.21.15.47:0/987320507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0120 0x7fc6140a84a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6140b2b00 tx=0x7fc60802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4b319bfe2164c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 1 -- 172.21.15.47:0/987320507 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608030860 con 0x7fc6140b0120 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: dump: 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:46.703 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 1 -- 172.21.15.47:0/987320507 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6080309c0 con 0x7fc6140b0120 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 1 -- 172.21.15.47:0/987320507 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608030d50 con 0x7fc6140b0120 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:46.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc611ffb640 10 monclient: dump: 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc61954e640 5 monclient: authenticate success, global_id 4254 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc61954e640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:46.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.701+0000 7fc61954e640 4 set_mon_vals no callback set 2024-03-30T07:16:46.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 10 monclient: shutdown 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 -- 172.21.15.47:0/987320507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0120 msgr2=0x7fc6140a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 --2- 172.21.15.47:0/987320507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0120 0x7fc6140a84a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc6140b2b00 tx=0x7fc60802b9a0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 -- 172.21.15.47:0/987320507 shutdown_connections 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 --2- 172.21.15.47:0/987320507 >> v2:172.21.15.52:3300/0 conn(0x7fc6140b0d20 0x7fc61414be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 --2- 172.21.15.47:0/987320507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0120 0x7fc6140a84a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 -- 172.21.15.47:0/987320507 >> 172.21.15.47:0/987320507 conn(0x7fc6140b70b0 msgr2=0x7fc6140b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 -- 172.21.15.47:0/987320507 shutdown_connections 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.702+0000 7fc61954e640 1 -- 172.21.15.47:0/987320507 wait complete. 2024-03-30T07:16:46.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 10 monclient: build_initial_monmap 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 10 monclient: monmap: 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 librados: starting msgr at 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 librados: starting objecter 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 Processor -- start 2024-03-30T07:16:46.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 -- start start 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 librados: setting wanted keys 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 1 librados: calling monclient init 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.703+0000 7fc61954e640 10 monclient: init 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 0x7fc6140ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-b con 0x7fc6140b0120 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0d20 0x7fc6140ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-a con 0x7fc6140b0d20 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): picked mon.noname-c con 0x7fc614144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140acd00 con 0x7fc6140b0d20 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:46.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140acfb0 con 0x7fc6140b0120 2024-03-30T07:16:46.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:46.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc614142920 con 0x7fc614144ab0 2024-03-30T07:16:46.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:46.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc61954e640 10 monclient(hunting): authenticate will time out at 1532.820567s 2024-03-30T07:16:46.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc612ffd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 0x7fc6140ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.704+0000 7fc6127fc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0d20 0x7fc6140ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54646/0 (socket says 172.21.15.47:54646) 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 1 -- 172.21.15.47:0/623388558 learned_addr learned my addr 172.21.15.47:0/623388558 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): get_auth_request con 0x7fc614144ab0 auth_method 0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): get_auth_request con 0x7fc6140b0d20 auth_method 0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): get_auth_request con 0x7fc6140b0120 auth_method 0 2024-03-30T07:16:46.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc612ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:46.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.705+0000 7fc6137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 -- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 msgr2=0x7fc6140ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 0x7fc6140ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 -- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0d20 msgr2=0x7fc6140ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0d20 0x7fc6140ac7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:46.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 10 monclient: _finish_hunting 0 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 monclient: found mon.noname-c 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc6137fe640 1 -- 172.21.15.47:0/623388558 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080094d0 con 0x7fc614144ab0 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.708+0000 7fc612ffd640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 0x7fc6140ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.710+0000 7fc6137fe640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009dd0 tx=0x7fc60c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.710+0000 7fc5fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc614144ab0 2024-03-30T07:16:46.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.710+0000 7fc5fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:46.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c014480 con 0x7fc614144ab0 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: dump: 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:46.718 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c00f040 con 0x7fc614144ab0 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 5 monclient: authenticate success, global_id 4205 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 10 monclient: _renew_subs 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6140ad500 con 0x7fc614144ab0 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c014840 con 0x7fc614144ab0 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc5fb7fe640 10 monclient: dump: 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:46.719 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc6117fa640 4 set_mon_vals no callback set 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 10 monclient: _renew_subs 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6141d9f90 con 0x7fc614144ab0 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.714+0000 7fc61954e640 1 librados: init done 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.715+0000 7fc61954e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:46.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.715+0000 7fc61954e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:46.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.715+0000 7fc61954e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:46.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.715+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6140ad160 con 0x7fc614144ab0 2024-03-30T07:16:46.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.739+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc60c0149a0 con 0x7fc614144ab0 2024-03-30T07:16:46.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.739+0000 7fc5fb7fe640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 0x7fc60006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:46.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.739+0000 7fc612ffd640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 0x7fc60006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:46.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.739+0000 7fc612ffd640 10 monclient: get_auth_request con 0x7fc60006c5e0 auth_method 0 2024-03-30T07:16:46.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.740+0000 7fc612ffd640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 0x7fc60006eaa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc608036000 tx=0x7fc608008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:46.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.740+0000 7fc5fb7fe640 10 client.4205.objecter ms_handle_connect 0x7fc60006c5e0 2024-03-30T07:16:46.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.741+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc60c059400 con 0x7fc614144ab0 2024-03-30T07:16:46.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.741+0000 7fc5fb7fe640 10 client.4205.objecter ms_dispatch 0x7fc614000b90 osd_map(23..23 src has 1..23) v4 2024-03-30T07:16:46.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.741+0000 7fc5fb7fe640 3 client.4205.objecter handle_osd_map got epochs [23,23] > 0 2024-03-30T07:16:46.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.741+0000 7fc5fb7fe640 3 client.4205.objecter handle_osd_map decoding full epoch 23 2024-03-30T07:16:46.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.741+0000 7fc5fb7fe640 20 client.4205.objecter dump_active .. 0 homeless 2024-03-30T07:16:46.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.997+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc60c055740 con 0x7fc614144ab0 2024-03-30T07:16:46.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.997+0000 7fc5fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:46.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:46.997+0000 7fc5fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:47.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:47.186+0000 7fc61954e640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-30T07:16:47.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:47.186+0000 7fc61954e640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-30T07:16:47.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:47.186+0000 7fc61954e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:47.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:47.186+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fc6140ad160 con 0x7fc614144ab0 2024-03-30T07:16:48.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.303+0000 7fc5fb7fe640 1 -- 172.21.15.47:0/623388558 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v24) v1 ==== 109+0+0 (secure 0 0 0) 0x7fc6140ad160 con 0x7fc614144ab0 2024-03-30T07:16:48.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.303+0000 7fc5fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-30T07:16:48.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.303+0000 7fc5fb7fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-03-30T07:16:48.305 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'rbd' created 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 10 librados: watch_flush enter 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 10 librados: watch_flush exit 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 20 client.4205.objecter shutdown clearing up homeless session... 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 10 client.4205.objecter successfully canceled tick 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 msgr2=0x7fc60006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 0x7fc60006eaa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc608036000 tx=0x7fc608008030 comp rx=0 tx=0).stop 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 10 monclient: shutdown 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 msgr2=0x7fc6141d9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:48.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009dd0 tx=0x7fc60c00c490 comp rx=0 tx=0).stop 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.305+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 shutdown_connections 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc60006c5e0 0x7fc60006eaa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc614144ab0 0x7fc6141d9a50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc6140b0d20 0x7fc6140ac7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 --2- 172.21.15.47:0/623388558 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc6140b0120 0x7fc6140ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 >> 172.21.15.47:0/623388558 conn(0x7fc6140b70b0 msgr2=0x7fc614068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 shutdown_connections 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 -- 172.21.15.47:0/623388558 wait complete. 2024-03-30T07:16:48.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.306+0000 7fc61954e640 1 librados: shutdown 2024-03-30T07:16:48.319 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-30T07:16:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 10 monclient: get_monmap_and_config 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 10 monclient: build_initial_monmap 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 10 monclient: monmap: 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:48.423889+0000 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:48.423889+0000 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:48.425 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 1 Processor -- start 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 1 -- start start 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.422+0000 7f7ea6dd0640 10 monclient: init 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f7ea00b43c0 0x7f7ea00a8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f7ea00b43c0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 0x7f7ea013ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f7ea013e890 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f7ea0144b80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea00a8da0 con 0x7f7ea013e890 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea00b47a0 con 0x7f7ea00b43c0 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea013f210 con 0x7f7ea0144b80 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.423+0000 7f7ea6dd0640 10 monclient(hunting): authenticate will time out at 1534.539742s 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 0x7f7ea013ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 0x7f7ea013ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50868/0 (socket says 172.21.15.47:50868) 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54652/0 (socket says 172.21.15.47:54652) 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 1 -- 172.21.15.47:0/3078956032 learned_addr learned my addr 172.21.15.47:0/3078956032 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ea013e890 auth_method 0 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 10 monclient(hunting): get_auth_request con 0x7f7ea0144b80 auth_method 0 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.424+0000 7f7ea536d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:48.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:48.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.425+0000 7f7ea4b6c640 1 --2- 172.21.15.47:0/3078956032 >> v2:172.21.15.52:3300/0 conn(0x7f7ea00b43c0 0x7f7ea00a8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea4b6c640 10 monclient(hunting): get_auth_request con 0x7f7ea00b43c0 auth_method 0 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea4b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea4b6c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea4b6c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 -- 172.21.15.47:0/3078956032 >> v2:172.21.15.52:3300/0 conn(0x7f7ea00b43c0 msgr2=0x7f7ea00a8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 --2- 172.21.15.47:0/3078956032 >> v2:172.21.15.52:3300/0 conn(0x7f7ea00b43c0 0x7f7ea00a8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 -- 172.21.15.47:0/3078956032 >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 msgr2=0x7f7ea013ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 --2- 172.21.15.47:0/3078956032 >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 0x7f7ea013ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 monclient: found mon.noname-b 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:48.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.426+0000 7f7ea536d640 1 -- 172.21.15.47:0/3078956032 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea0145500 con 0x7f7ea0144b80 2024-03-30T07:16:48.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.429+0000 7f7ea536d640 1 --2- 172.21.15.47:0/3078956032 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00b81c0 tx=0x7f7e9802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70d9ca5411a6f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 1 -- 172.21.15.47:0/3078956032 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9802cb30 con 0x7f7ea0144b80 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: dump: 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:48.435 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 1 -- 172.21.15.47:0/3078956032 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9802cc90 con 0x7f7ea0144b80 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 1 -- 172.21.15.47:0/3078956032 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98034440 con 0x7f7ea0144b80 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 5 monclient: authenticate success, global_id 4208 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 4 set_mon_vals no callback set 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7e9f7fe640 10 monclient: dump: 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:48.437 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 10 monclient: shutdown 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/3078956032 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 msgr2=0x7f7ea0144fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/3078956032 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7ea00b81c0 tx=0x7f7e9802bd80 comp rx=0 tx=0).stop 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/3078956032 shutdown_connections 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/3078956032 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea0144b80 0x7f7ea0144fc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/3078956032 >> v2:172.21.15.47:3300/0 conn(0x7f7ea013e890 0x7f7ea013ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/3078956032 >> v2:172.21.15.52:3300/0 conn(0x7f7ea00b43c0 0x7f7ea00a8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/3078956032 >> 172.21.15.47:0/3078956032 conn(0x7f7ea0071930 msgr2=0x7f7ea0071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.433+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/3078956032 shutdown_connections 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.434+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/3078956032 wait complete. 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.434+0000 7f7ea6dd0640 10 monclient: build_initial_monmap 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.434+0000 7f7ea6dd0640 10 monclient: monmap: 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 librados: starting msgr at 2024-03-30T07:16:48.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 librados: starting objecter 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 Processor -- start 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 -- start start 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 librados: setting wanted keys 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 1 librados: calling monclient init 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.435+0000 7f7ea6dd0640 10 monclient: init 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ea00b43c0 0x7f7ea00b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f7ea00b43c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f7ea013e890 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ea0144b80 0x7f7ea01da400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f7ea0144b80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:48.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea01da940 con 0x7f7ea00b43c0 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea01d45d0 con 0x7f7ea0144b80 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea01dab80 con 0x7f7ea013e890 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea6dd0640 10 monclient(hunting): authenticate will time out at 1534.552223s 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea536d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ea0144b80 0x7f7ea01da400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea4b6c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ea00b43c0 0x7f7ea00b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54656/0 (socket says 172.21.15.47:54656) 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 1 -- 172.21.15.47:0/4197261351 learned_addr learned my addr 172.21.15.47:0/4197261351 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 10 monclient(hunting): get_auth_request con 0x7f7ea013e890 auth_method 0 2024-03-30T07:16:48.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea536d640 10 monclient(hunting): get_auth_request con 0x7f7ea0144b80 auth_method 0 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea536d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea536d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.436+0000 7f7ea536d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): get_auth_request con 0x7f7ea00b43c0 auth_method 0 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:48.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea536d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.437+0000 7f7ea4b6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 -- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ea0144b80 msgr2=0x7f7ea01da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ea0144b80 0x7f7ea01da400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 -- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ea00b43c0 msgr2=0x7f7ea00b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ea00b43c0 0x7f7ea00b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 monclient: found mon.noname-c 2024-03-30T07:16:48.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.440+0000 7f7e9ffff640 1 -- 172.21.15.47:0/4197261351 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98009a70 con 0x7f7ea013e890 2024-03-30T07:16:48.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.442+0000 7f7e9ffff640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00e8b0 tx=0x7f7e8c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:48.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.442+0000 7f7e9d7fa640 10 client.?.objecter ms_handle_connect 0x7f7ea013e890 2024-03-30T07:16:48.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.442+0000 7f7e9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:48.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00c940 con 0x7f7ea013e890 2024-03-30T07:16:48.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: dump: 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:48.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00caa0 con 0x7f7ea013e890 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 5 monclient: authenticate success, global_id 4211 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 10 monclient: _renew_subs 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea0141c10 con 0x7f7ea013e890 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00ce30 con 0x7f7ea013e890 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9d7fa640 10 monclient: dump: 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:48.452 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7e9effd640 4 set_mon_vals no callback set 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 10 monclient: _renew_subs 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea01420f0 con 0x7f7ea013e890 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.446+0000 7f7ea6dd0640 1 librados: init done 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.447+0000 7f7ea6dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.447+0000 7f7ea6dd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:48.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.447+0000 7f7ea6dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.447+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea013d3c0 con 0x7f7ea013e890 2024-03-30T07:16:48.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.715+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e8c013470 con 0x7f7ea013e890 2024-03-30T07:16:48.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.716+0000 7f7e9d7fa640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 0x7f7e8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:48.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.716+0000 7f7ea4b6c640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 0x7f7e8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:48.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.716+0000 7f7ea4b6c640 10 monclient: get_auth_request con 0x7f7e8406c5e0 auth_method 0 2024-03-30T07:16:48.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.717+0000 7f7e9d7fa640 10 client.4211.objecter ms_handle_connect 0x7f7e8406c5e0 2024-03-30T07:16:48.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.717+0000 7f7ea4b6c640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 0x7f7e8406eaa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006fd0 tx=0x7f7e94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:48.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f7e8c058b40 con 0x7f7ea013e890 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 10 client.4211.objecter ms_dispatch 0x7f7ea0000b90 osd_map(24..24 src has 1..24) v4 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 3 client.4211.objecter handle_osd_map got epochs [24,24] > 0 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 3 client.4211.objecter handle_osd_map decoding full epoch 24 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7e8c054e80 con 0x7f7ea013e890 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:48.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.731+0000 7f7e9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:48.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.943+0000 7f7ea6dd0640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-30T07:16:48.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.943+0000 7f7ea6dd0640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-30T07:16:48.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.943+0000 7f7ea6dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:48.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:48.943+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7ea00b1170 con 0x7f7ea013e890 2024-03-30T07:16:49.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.362+0000 7f7e9d7fa640 1 -- 172.21.15.47:0/4197261351 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7e8c0210c0 con 0x7f7ea013e890 2024-03-30T07:16:49.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.362+0000 7f7e9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-30T07:16:49.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.362+0000 7f7e9d7fa640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-03-30T07:16:49.364 INFO:teuthology.orchestra.run.smithi047.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 10 librados: watch_flush enter 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 10 librados: watch_flush exit 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 10 client.4211.objecter successfully canceled tick 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 msgr2=0x7f7e8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 0x7f7e8406eaa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006fd0 tx=0x7f7e94008040 comp rx=0 tx=0).stop 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 10 monclient: shutdown 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 msgr2=0x7f7ea01d9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.364+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00e8b0 tx=0x7f7e8c00ed70 comp rx=0 tx=0).stop 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 shutdown_connections 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7e8406c5e0 0x7f7e8406eaa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7ea0144b80 0x7f7ea01da400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ea013e890 0x7f7ea01d9ec0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 --2- 172.21.15.47:0/4197261351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ea00b43c0 0x7f7ea00b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 >> 172.21.15.47:0/4197261351 conn(0x7f7ea0071930 msgr2=0x7f7ea013ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 shutdown_connections 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 -- 172.21.15.47:0/4197261351 wait complete. 2024-03-30T07:16:49.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.365+0000 7f7ea6dd0640 1 librados: shutdown 2024-03-30T07:16:49.379 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-30T07:16:49.380 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T07:16:49.380 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-30T07:16:49.380 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-30T07:16:49.381 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.492+0000 7fedccc6a640 10 monclient: get_monmap_and_config 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.492+0000 7fedccc6a640 10 monclient: build_initial_monmap 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.492+0000 7fedccc6a640 10 monclient: monmap: 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:49.494140+0000 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:49.494140+0000 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.494 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:49.495 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:49.495 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:49.495 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:49.495 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:49.495 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 1 Processor -- start 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 1 -- start start 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient: init 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fedc80b4280 0x7fedc80a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-a con 0x7fedc80b4280 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedc813e930 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.493+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedc813eda0 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc80aa3c0 con 0x7fedc813e930 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc80aa630 con 0x7fedc813eda0 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc80b4660 con 0x7fedc80b4280 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedccc6a640 10 monclient(hunting): authenticate will time out at 1535.610164s 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedc6d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fedc813eda0 0x7fedc8141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56198/0 (socket says 172.21.15.47:56198) 2024-03-30T07:16:49.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.494+0000 7fedc6d9d640 1 -- 172.21.15.47:0/93124714 learned_addr learned my addr 172.21.15.47:0/93124714 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791626812 0 0) 0x7fedc80aa630 con 0x7fedc813eda0 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 10 monclient(hunting): _negotiate 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 10 monclient(hunting): my global_id is 4273 2024-03-30T07:16:49.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.498+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0003520 con 0x7fedc813eda0 2024-03-30T07:16:49.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.501+0000 7fedc659c640 1 --2- 172.21.15.47:0/93124714 >> v2:172.21.15.47:3300/0 conn(0x7fedc80b4280 0x7fedc80a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc659c640 10 monclient(hunting): get_auth_request con 0x7fedc80b4280 auth_method 0 2024-03-30T07:16:49.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc659c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.1 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594866430 0 0) 0x7fedb0003520 con 0x7fedc813eda0 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 >> v2:172.21.15.47:3300/0 conn(0x7fedc80b4280 msgr2=0x7fedc80a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 --2- 172.21.15.47:0/93124714 >> v2:172.21.15.47:3300/0 conn(0x7fedc80b4280 0x7fedc80a8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 >> v1:172.21.15.47:6789/0 conn(0x7fedc813e930 legacy=0x7fedc8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _finish_hunting 0 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 monclient: found mon.noname-c-legacy 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.52:6789/0 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc80a9210 con 0x7fedc813eda0 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _finish_auth 0 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:49.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.1 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7fedc00019f0 con 0x7fedc813eda0 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: dump: 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] but i'm connected to v1:172.21.15.52:6789/0, reconnecting 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 >> v1:172.21.15.52:6789/0 conn(0x7fedc813eda0 legacy=0x7fedc8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 0x7fedb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): picked mon.c con 0x7fedb0003970 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedb00047c0 0x7fedb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): picked mon.a con 0x7fedb00047c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb00052a0 0x7fedb000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): picked mon.b con 0x7fedb00052a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc80b4660 con 0x7fedb00047c0 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000c170 con 0x7fedb00052a0 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000c330 con 0x7fedb0003970 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.502+0000 7fedc559a640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.503+0000 7fedc659c640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 0x7fedb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.503+0000 7fedc659c640 10 monclient(hunting): get_auth_request con 0x7fedb0003970 auth_method 0 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.503+0000 7fedc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.503+0000 7fedc659c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.503+0000 7fedc659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.504+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.504+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:49.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.504+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.504+0000 7fedc6d9d640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb00052a0 0x7fedb000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): get_auth_request con 0x7fedb00052a0 auth_method 0 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.505+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 10 monclient(hunting): handle_auth_done global_id 4273 payload 293 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 -- 172.21.15.47:0/93124714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb00052a0 msgr2=0x7fedb000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb00052a0 0x7fedb000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 -- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedb00047c0 msgr2=0x7fedb0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:49.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedb00047c0 0x7fedb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc6d9d640 1 -- 172.21.15.47:0/93124714 reap_dead start 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 monclient: found mon.c 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 1 -- 172.21.15.47:0/93124714 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc80a9210 con 0x7fedb0003970 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 10 monclient: _finish_auth 0 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 10 monclient: _check_auth_tickets 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedc659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedccc6a640 5 monclient: authenticate success, global_id 4273 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.507+0000 7fedccc6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:49.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.509+0000 7fedc659c640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 0x7fedb0003d90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fedc8069c70 tx=0x7fedb00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b3153bbfa5526c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:49.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.513+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8034620 con 0x7fedb0003970 2024-03-30T07:16:49.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:49.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:49.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: dump: 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb8034780 con 0x7fedb0003970 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:49.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 1 -- 172.21.15.47:0/93124714 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb8034af0 con 0x7fedb0003970 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 4 set_mon_vals no callback set 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedc559a640 10 monclient: dump: 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:49.517 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 10 monclient: shutdown 2024-03-30T07:16:49.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:49.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 -- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 msgr2=0x7fedb0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 0x7fedb0003d90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fedc8069c70 tx=0x7fedb00057f0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 -- 172.21.15.47:0/93124714 shutdown_connections 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 --2- 172.21.15.47:0/93124714 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedb0003970 0x7fedb0003d90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 -- 172.21.15.47:0/93124714 >> 172.21.15.47:0/93124714 conn(0x7fedc8065e00 msgr2=0x7fedc8066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.514+0000 7fedccc6a640 1 -- 172.21.15.47:0/93124714 shutdown_connections 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.515+0000 7fedccc6a640 1 -- 172.21.15.47:0/93124714 wait complete. 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.515+0000 7fedccc6a640 10 monclient: build_initial_monmap 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.515+0000 7fedccc6a640 10 monclient: monmap: 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.521 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 librados: starting msgr at 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 librados: starting objecter 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 Processor -- start 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 -- start start 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 librados: setting wanted keys 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 librados: calling monclient init 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient: init 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:49.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb0003970 0x7fedc81433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-b con 0x7fedb0003970 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-c con 0x7fedc8142c20 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedc8135990 0x7fedc81d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): picked mon.noname-a con 0x7fedc8135990 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8145040 con 0x7fedc8135990 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8142970 con 0x7fedb0003970 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc813dd30 con 0x7fedc8142c20 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.516+0000 7fedccc6a640 10 monclient(hunting): authenticate will time out at 1535.632997s 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc659c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb0003970 0x7fedc81433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc6d9d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedc8135990 0x7fedc81d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54674/0 (socket says 172.21.15.47:54674) 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 1 -- 172.21.15.47:0/1606884092 learned_addr learned my addr 172.21.15.47:0/1606884092 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 10 monclient(hunting): get_auth_request con 0x7fedc8142c20 auth_method 0 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc6d9d640 10 monclient(hunting): get_auth_request con 0x7fedc8135990 auth_method 0 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc659c640 10 monclient(hunting): get_auth_request con 0x7fedb0003970 auth_method 0 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc659c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:49.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.517+0000 7fedc659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:49.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.518+0000 7fedc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:49.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2024-03-30T07:16:49.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 -- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb0003970 msgr2=0x7fedc81433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb0003970 0x7fedc81433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 -- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedc8135990 msgr2=0x7fedc81d5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:49.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedc8135990 0x7fedc81d5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:49.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 monclient: found mon.noname-c 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.521+0000 7fedc5d9b640 1 -- 172.21.15.47:0/1606884092 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8145460 con 0x7fedc8142c20 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.523+0000 7fedc5d9b640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c500 tx=0x7fedbc00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.523+0000 7fedb6ffd640 10 client.?.objecter ms_handle_connect 0x7fedc8142c20 2024-03-30T07:16:49.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.523+0000 7fedb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc0052d0 con 0x7fedc8142c20 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: dump: 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.531 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedbc005450 con 0x7fedc8142c20 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedccc6a640 5 monclient: authenticate success, global_id 4214 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedccc6a640 10 monclient: _renew_subs 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedccc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc81d64e0 con 0x7fedc8142c20 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedc4d99640 4 set_mon_vals no callback set 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.528+0000 7fedccc6a640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:49.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedbc003850 con 0x7fedc8142c20 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedb6ffd640 10 monclient: dump: 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedccc6a640 10 monclient: _renew_subs 2024-03-30T07:16:49.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedccc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc81d6980 con 0x7fedc8142c20 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.529+0000 7fedccc6a640 1 librados: init done 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.530+0000 7fedccc6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.530+0000 7fedccc6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.530+0000 7fedccc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:49.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.530+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc804e030 con 0x7fedc8142c20 2024-03-30T07:16:49.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.806+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fedbc0039b0 con 0x7fedc8142c20 2024-03-30T07:16:49.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.807+0000 7fedb6ffd640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 0x7feda806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:49.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.807+0000 7fedc659c640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 0x7feda806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:49.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.807+0000 7fedc659c640 10 monclient: get_auth_request con 0x7feda806c590 auth_method 0 2024-03-30T07:16:49.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.808+0000 7fedc659c640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 0x7feda806ea50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fedc804ede0 tx=0x7fedb8036040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:49.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.808+0000 7fedb6ffd640 10 client.4214.objecter ms_handle_connect 0x7feda806c590 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fedbc0580a0 con 0x7fedc8142c20 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 10 client.4214.objecter ms_dispatch 0x7fedc8000b90 osd_map(25..25 src has 1..25) v4 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 3 client.4214.objecter handle_osd_map got epochs [25,25] > 0 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 3 client.4214.objecter handle_osd_map decoding full epoch 25 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fedbc0543e0 con 0x7fedc8142c20 2024-03-30T07:16:49.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:49.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:49.826+0000 7fedb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:50.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.099+0000 7fedccc6a640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-03-30T07:16:50.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.099+0000 7fedccc6a640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-30T07:16:50.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.099+0000 7fedccc6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:50.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.099+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fedc8143e50 con 0x7fedc8142c20 2024-03-30T07:16:50.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.251+0000 7fedb6ffd640 1 -- 172.21.15.47:0/1606884092 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170936 (secure 0 0 0) 0x7fedbc057a00 con 0x7fedc8142c20 2024-03-30T07:16:50.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.251+0000 7fedb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-30T07:16:50.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.251+0000 7fedb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:50.256 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 10 librados: watch_flush enter 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 10 librados: watch_flush exit 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 10 client.4214.objecter successfully canceled tick 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 msgr2=0x7feda806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 0x7feda806ea50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fedc804ede0 tx=0x7fedb8036040 comp rx=0 tx=0).stop 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 10 monclient: shutdown 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 msgr2=0x7fedc8135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c500 tx=0x7fedbc00c9c0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 shutdown_connections 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feda806c590 0x7feda806ea50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fedc8135990 0x7fedc81d5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fedc8142c20 0x7fedc8135450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 --2- 172.21.15.47:0/1606884092 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fedb0003970 0x7fedc81433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 >> 172.21.15.47:0/1606884092 conn(0x7fedc8065e00 msgr2=0x7fedc81365f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 shutdown_connections 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 -- 172.21.15.47:0/1606884092 wait complete. 2024-03-30T07:16:50.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.254+0000 7fedccc6a640 1 librados: shutdown 2024-03-30T07:16:50.267 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":34694},{"type":"v1","addr":"172.21.15.52:6801","nonce":34694}]},"active_addr":"172.21.15.52:6801/34694","active_change":"2024-03-30T07:15:49.563806+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2831416639}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1423725251}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2373324101}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":768834739}]}]} 2024-03-30T07:16:50.272 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-30T07:16:50.272 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-30T07:16:50.273 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.372+0000 7f07895a2640 10 monclient: get_monmap_and_config 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.372+0000 7f07895a2640 10 monclient: build_initial_monmap 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.372+0000 7f07895a2640 10 monclient: monmap: 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:50.373186+0000 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:50.373186+0000 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:50.374 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 Processor -- start 2024-03-30T07:16:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 -- start start 2024-03-30T07:16:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient: init 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07840a8710 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f07840b3500 0x7f07840b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-a con 0x7f07840b3500 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0784144910 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07840a8af0 con 0x7f07840a8710 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078413f000 con 0x7f07840b3500 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07840b43c0 con 0x7f0784144910 2024-03-30T07:16:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f07895a2640 10 monclient(hunting): authenticate will time out at 1536.489004s 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54676/0 (socket says 172.21.15.47:54676) 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 1 -- 172.21.15.47:0/3090547894 learned_addr learned my addr 172.21.15.47:0/3090547894 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 10 monclient(hunting): get_auth_request con 0x7f0784144910 auth_method 0 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.373+0000 7f0788da1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.374+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.374+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.374+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 1 -- 172.21.15.47:0/3090547894 >> v2:172.21.15.47:3300/0 conn(0x7f07840b3500 msgr2=0x7f07840b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 1 --2- 172.21.15.47:0/3090547894 >> v2:172.21.15.47:3300/0 conn(0x7f07840b3500 0x7f07840b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 1 -- 172.21.15.47:0/3090547894 >> v1:172.21.15.47:6789/0 conn(0x7f07840a8710 legacy=0x7f07840b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 10 monclient: _finish_hunting 0 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 1 monclient: found mon.noname-b 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.376+0000 7f0788da1640 1 -- 172.21.15.47:0/3090547894 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07840b14a0 con 0x7f0784144910 2024-03-30T07:16:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.378+0000 7f0788da1640 1 --2- 172.21.15.47:0/3090547894 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f077c009e50 tx=0x7f077c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea84b57f33403667 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 1 -- 172.21.15.47:0/3090547894 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c02ca70 con 0x7f0784144910 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: dump: 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 1 -- 172.21.15.47:0/3090547894 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c02cbd0 con 0x7f0784144910 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 1 -- 172.21.15.47:0/3090547894 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c0343f0 con 0x7f0784144910 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 5 monclient: authenticate success, global_id 4217 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 4 set_mon_vals no callback set 2024-03-30T07:16:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f0782ffd640 10 monclient: dump: 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 10 monclient: shutdown 2024-03-30T07:16:50.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 -- 172.21.15.47:0/3090547894 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 msgr2=0x7f07840afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 --2- 172.21.15.47:0/3090547894 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f077c009e50 tx=0x7f077c02bed0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 -- 172.21.15.47:0/3090547894 shutdown_connections 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 --2- 172.21.15.47:0/3090547894 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0784144910 0x7f07840afd90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 --2- 172.21.15.47:0/3090547894 >> v2:172.21.15.47:3300/0 conn(0x7f07840b3500 0x7f07840b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 -- 172.21.15.47:0/3090547894 >> 172.21.15.47:0/3090547894 conn(0x7f0784065bc0 msgr2=0x7f0784065fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 -- 172.21.15.47:0/3090547894 shutdown_connections 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.385+0000 7f07895a2640 1 -- 172.21.15.47:0/3090547894 wait complete. 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.386+0000 7f07895a2640 10 monclient: build_initial_monmap 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.386+0000 7f07895a2640 10 monclient: monmap: 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:50.391 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:50.392 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.392 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:50.392 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:50.392 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:50.392 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.386+0000 7f07895a2640 1 librados: starting msgr at 2024-03-30T07:16:50.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.386+0000 7f07895a2640 1 librados: starting objecter 2024-03-30T07:16:50.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.386+0000 7f07895a2640 1 Processor -- start 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 -- start start 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 librados: setting wanted keys 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 librados: calling monclient init 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient: init 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:50.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:50.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07840a8710 0x7f0784141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-a con 0x7f07840a8710 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 0x7f078413b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-c con 0x7f07840b3500 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 0x7f078413cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0784144910 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078413d570 con 0x7f07840a8710 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07840ab0c0 con 0x7f0784144910 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07840ab330 con 0x7f07840b3500 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.387+0000 7f07895a2640 10 monclient(hunting): authenticate will time out at 1536.503869s 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 0x7f078413b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:50.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f0788da1640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 0x7f078413cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 0x7f078413b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54684/0 (socket says 172.21.15.47:54684) 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 1 -- 172.21.15.47:0/392195175 learned_addr learned my addr 172.21.15.47:0/392195175 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 10 monclient(hunting): get_auth_request con 0x7f07840b3500 auth_method 0 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f07837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:50.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f0788da1640 10 monclient(hunting): get_auth_request con 0x7f0784144910 auth_method 0 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f0788da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f0788da1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.388+0000 7f0788da1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f07837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.389+0000 7f0788da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 -- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 msgr2=0x7f078413b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 0x7f078413b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 -- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07840a8710 msgr2=0x7f0784141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07840a8710 0x7f0784141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:50.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 10 monclient: _finish_hunting 0 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 monclient: found mon.noname-b 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.392+0000 7f0788da1640 1 -- 172.21.15.47:0/392195175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c009b30 con 0x7f0784144910 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.395+0000 7f0788da1640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 0x7f078413cbf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f077c032000 tx=0x7f077c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.395+0000 7f0780ff9640 10 client.?.objecter ms_handle_connect 0x7f0784144910 2024-03-30T07:16:50.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.395+0000 7f0780ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:50.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c03e450 con 0x7f0784144910 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: dump: 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: _finish_auth 0 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c03e5b0 con 0x7f0784144910 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c03e760 con 0x7f0784144910 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 5 monclient: authenticate success, global_id 4279 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0780ff9640 10 monclient: dump: 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:50.454 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 10 monclient: _renew_subs 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078413d130 con 0x7f0784144910 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f0781ffb640 4 set_mon_vals no callback set 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 10 monclient: _renew_subs 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078413db70 con 0x7f0784144910 2024-03-30T07:16:50.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.450+0000 7f07895a2640 1 librados: init done 2024-03-30T07:16:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.451+0000 7f07895a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.451+0000 7f07895a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.451+0000 7f07895a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:50.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.451+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078413d6b0 con 0x7f0784144910 2024-03-30T07:16:50.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.728+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f077c03e8c0 con 0x7f0784144910 2024-03-30T07:16:50.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.728+0000 7f0780ff9640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 0x7f077006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:50.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.728+0000 7f0783fff640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 0x7f077006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:50.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.728+0000 7f0783fff640 10 monclient: get_auth_request con 0x7f077006c5e0 auth_method 0 2024-03-30T07:16:50.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.729+0000 7f0780ff9640 10 client.4279.objecter ms_handle_connect 0x7f077006c5e0 2024-03-30T07:16:50.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.729+0000 7f0783fff640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 0x7f077006eaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0774001560 tx=0x7f0774007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:50.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.731+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f077c07b9a0 con 0x7f0784144910 2024-03-30T07:16:50.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.731+0000 7f0780ff9640 10 client.4279.objecter ms_dispatch 0x7f0784000b90 osd_map(25..25 src has 1..25) v4 2024-03-30T07:16:50.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.731+0000 7f0780ff9640 3 client.4279.objecter handle_osd_map got epochs [25,25] > 0 2024-03-30T07:16:50.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.731+0000 7f0780ff9640 3 client.4279.objecter handle_osd_map decoding full epoch 25 2024-03-30T07:16:50.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.732+0000 7f0780ff9640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-30T07:16:50.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.749+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f077c0775e0 con 0x7f0784144910 2024-03-30T07:16:50.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.749+0000 7f0780ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:50.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.749+0000 7f0780ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:50.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.933+0000 7f07895a2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:50.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.933+0000 7f07895a2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:50.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.933+0000 7f07895a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:50.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:50.933+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f078413d6b0 con 0x7f0784144910 2024-03-30T07:16:51.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.039+0000 7f0780ff9640 1 -- 172.21.15.47:0/392195175 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14159 (secure 0 0 0) 0x7f078413d6b0 con 0x7f0784144910 2024-03-30T07:16:51.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.039+0000 7f0780ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:51.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.039+0000 7f0780ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:51.040 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:51.041 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":26,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:16:50.371801+0000","last_up_change":"2024-03-30T07:16:44.160131+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T07:16:32.182687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T07:16:47.206094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":35205},{"type":"v1","addr":"172.21.15.47:6801","nonce":35205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":35205},{"type":"v1","addr":"172.21.15.47:6803","nonce":35205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":35205},{"type":"v1","addr":"172.21.15.47:6807","nonce":35205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":35205},{"type":"v1","addr":"172.21.15.47:6805","nonce":35205}]},"public_addr":"172.21.15.47:6801/35205","cluster_addr":"172.21.15.47:6803/35205","heartbeat_back_addr":"172.21.15.47:6807/35205","heartbeat_front_addr":"172.21.15.47:6805/35205","state":["exists","up"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":35204},{"type":"v1","addr":"172.21.15.47:6809","nonce":35204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":35204},{"type":"v1","addr":"172.21.15.47:6813","nonce":35204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":35204},{"type":"v1","addr":"172.21.15.47:6820","nonce":35204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":35204},{"type":"v1","addr":"172.21.15.47:6816","nonce":35204}]},"public_addr":"172.21.15.47:6809/35204","cluster_addr":"172.21.15.47:6813/35204","heartbeat_back_addr":"172.21.15.47:6820/35204","heartbeat_front_addr":"172.21.15.47:6816/35204","state":["exists","up"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":35207},{"type":"v1","addr":"172.21.15.47:6825","nonce":35207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":35207},{"type":"v1","addr":"172.21.15.47:6827","nonce":35207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":35207},{"type":"v1","addr":"172.21.15.47:6831","nonce":35207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":35207},{"type":"v1","addr":"172.21.15.47:6829","nonce":35207}]},"public_addr":"172.21.15.47:6825/35207","cluster_addr":"172.21.15.47:6827/35207","heartbeat_back_addr":"172.21.15.47:6831/35207","heartbeat_front_addr":"172.21.15.47:6829/35207","state":["exists","up"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":35313},{"type":"v1","addr":"172.21.15.52:6819","nonce":35313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":35313},{"type":"v1","addr":"172.21.15.52:6821","nonce":35313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":35313},{"type":"v1","addr":"172.21.15.52:6825","nonce":35313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":35313},{"type":"v1","addr":"172.21.15.52:6823","nonce":35313}]},"public_addr":"172.21.15.52:6819/35313","cluster_addr":"172.21.15.52:6821/35313","heartbeat_back_addr":"172.21.15.52:6825/35313","heartbeat_front_addr":"172.21.15.52:6823/35313","state":["exists","up"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":35314},{"type":"v1","addr":"172.21.15.52:6829","nonce":35314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6829/35314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":35315},{"type":"v1","addr":"172.21.15.52:6807","nonce":35315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6807/35315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":35316},{"type":"v1","addr":"172.21.15.52:6806","nonce":35316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":35316},{"type":"v1","addr":"172.21.15.52:6812","nonce":35316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":35316},{"type":"v1","addr":"172.21.15.52:6817","nonce":35316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":35316},{"type":"v1","addr":"172.21.15.52:6815","nonce":35316}]},"public_addr":"172.21.15.52:6806/35316","cluster_addr":"172.21.15.52:6812/35316","heartbeat_back_addr":"172.21.15.52:6817/35316","heartbeat_front_addr":"172.21.15.52:6815/35316","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.613520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.651813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.749978+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.080129+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.393136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.240245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.041599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:27.954523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:51.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 10 librados: watch_flush enter 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 10 librados: watch_flush exit 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 10 client.4279.objecter successfully canceled tick 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 msgr2=0x7f077006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 0x7f077006eaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0774001560 tx=0x7f0774007990 comp rx=0 tx=0).stop 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 10 monclient: shutdown 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 msgr2=0x7f078413cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 0x7f078413cbf0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f077c032000 tx=0x7f077c004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 shutdown_connections 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f077006c5e0 0x7f077006eaa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f0784144910 0x7f078413cbf0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f07840b3500 0x7f078413b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 --2- 172.21.15.47:0/392195175 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f07840a8710 0x7f0784141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 >> 172.21.15.47:0/392195175 conn(0x7f0784065bc0 msgr2=0x7f07840b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 shutdown_connections 2024-03-30T07:16:51.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.041+0000 7f07895a2640 1 -- 172.21.15.47:0/392195175 wait complete. 2024-03-30T07:16:51.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.042+0000 7f07895a2640 1 librados: shutdown 2024-03-30T07:16:51.053 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-30T07:16:51.053 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.160+0000 7f9136b4e640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.160+0000 7f9136b4e640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.160+0000 7f9136b4e640 10 monclient: monmap: 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.161037+0000 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.161037+0000 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.161 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.162 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.162 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.162 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 Processor -- start 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 -- start start 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient: init 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 0x7f91300a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-b con 0x7f9130071cb0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9130144a80 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f913013ec10 0x7f913014be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-c con 0x7f913013ec10 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91300a88e0 con 0x7f9130144a80 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9130072090 con 0x7f913013ec10 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91300a92b0 con 0x7f9130071cb0 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.161+0000 7f9136b4e640 10 monclient(hunting): authenticate will time out at 1537.276902s 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f912ffff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9130144a80 0x7f913013e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40746/0 (socket says 172.21.15.47:40746) 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f912ffff640 1 -- 172.21.15.47:0/336225559 learned_addr learned my addr 172.21.15.47:0/336225559 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91348ea640 1 --2- 172.21.15.47:0/336225559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 0x7f91300a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91348ea640 10 monclient(hunting): get_auth_request con 0x7f9130071cb0 auth_method 0 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91348ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91348ea640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91348ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.162+0000 7f91350eb640 1 --2- 172.21.15.47:0/336225559 >> v2:172.21.15.52:3300/0 conn(0x7f913013ec10 0x7f913014be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): get_auth_request con 0x7f913013ec10 auth_method 0 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.163+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 1 -- 172.21.15.47:0/336225559 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229341753 0 0) 0x7f91300a88e0 con 0x7f9130144a80 2024-03-30T07:16:51.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 10 monclient(hunting): _negotiate 2024-03-30T07:16:51.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 10 monclient(hunting): my global_id is 4275 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f912f7fe640 1 -- 172.21.15.47:0/336225559 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c003520 con 0x7f9130144a80 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f91348ea640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f91348ea640 1 -- 172.21.15.47:0/336225559 >> v2:172.21.15.52:3300/0 conn(0x7f913013ec10 msgr2=0x7f913014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.164+0000 7f91348ea640 1 --2- 172.21.15.47:0/336225559 >> v2:172.21.15.52:3300/0 conn(0x7f913013ec10 0x7f913014be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 1 -- 172.21.15.47:0/336225559 >> v1:172.21.15.47:6789/0 conn(0x7f9130144a80 legacy=0x7f913013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 1 monclient: found mon.noname-b 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.165+0000 7f91348ea640 1 -- 172.21.15.47:0/336225559 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91300aa350 con 0x7f9130071cb0 2024-03-30T07:16:51.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.167+0000 7f91348ea640 1 --2- 172.21.15.47:0/336225559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 0x7f91300a8d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9128009d90 tx=0x7f912802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c2454cab9e5c8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 1 -- 172.21.15.47:0/336225559 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912802cbc0 con 0x7f9130071cb0 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: dump: 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.172 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 1 -- 172.21.15.47:0/336225559 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912802cd20 con 0x7f9130071cb0 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 1 -- 172.21.15.47:0/336225559 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128034660 con 0x7f9130071cb0 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 5 monclient: authenticate success, global_id 4223 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:51.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 4 set_mon_vals no callback set 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f912f7fe640 10 monclient: dump: 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.174 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 10 monclient: shutdown 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 -- 172.21.15.47:0/336225559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 msgr2=0x7f91300a8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 --2- 172.21.15.47:0/336225559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 0x7f91300a8d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9128009d90 tx=0x7f912802be10 comp rx=0 tx=0).stop 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 -- 172.21.15.47:0/336225559 shutdown_connections 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 --2- 172.21.15.47:0/336225559 >> v2:172.21.15.52:3300/0 conn(0x7f913013ec10 0x7f913014be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 --2- 172.21.15.47:0/336225559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9130071cb0 0x7f91300a8d70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 -- 172.21.15.47:0/336225559 >> 172.21.15.47:0/336225559 conn(0x7f91300b73d0 msgr2=0x7f91300b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 -- 172.21.15.47:0/336225559 shutdown_connections 2024-03-30T07:16:51.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.171+0000 7f9136b4e640 1 -- 172.21.15.47:0/336225559 wait complete. 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 10 monclient: monmap: 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 librados: starting msgr at 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 librados: starting objecter 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 Processor -- start 2024-03-30T07:16:51.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 -- start start 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 librados: setting wanted keys 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 1 librados: calling monclient init 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.172+0000 7f9136b4e640 10 monclient: init 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-a con 0x7f9130071cb0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f913013ec10 0x7f91300aad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-c con 0x7f913013ec10 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9130144a80 0x7f913013f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): picked mon.noname-b con 0x7f9130144a80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9130141200 con 0x7f9130071cb0 2024-03-30T07:16:51.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91301d9bf0 con 0x7f9130144a80 2024-03-30T07:16:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913013f840 con 0x7f913013ec10 2024-03-30T07:16:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.173+0000 7f9136b4e640 10 monclient(hunting): authenticate will time out at 1537.289024s 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f912ffff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f913013ec10 0x7f91300aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91350eb640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9130144a80 0x7f913013f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50910/0 (socket says 172.21.15.47:50910) 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 1 -- 172.21.15.47:0/606335432 learned_addr learned my addr 172.21.15.47:0/606335432 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 10 monclient(hunting): get_auth_request con 0x7f9130071cb0 auth_method 0 2024-03-30T07:16:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f91348ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f912ffff640 10 monclient(hunting): get_auth_request con 0x7f913013ec10 auth_method 0 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f912ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f912ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.174+0000 7f912ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): get_auth_request con 0x7f9130144a80 auth_method 0 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91348ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f912ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f912ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f912ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.175+0000 7f91350eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 -- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f913013ec10 msgr2=0x7f91300aad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f913013ec10 0x7f91300aad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 -- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9130144a80 msgr2=0x7f913013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9130144a80 0x7f913013f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 monclient: found mon.noname-a 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.178+0000 7f91348ea640 1 -- 172.21.15.47:0/606335432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009a70 con 0x7f9130071cb0 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.181+0000 7f91348ea640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9128037040 tx=0x7f9128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.181+0000 7f912d7fa640 10 client.?.objecter ms_handle_connect 0x7f9130071cb0 2024-03-30T07:16:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.181+0000 7f912d7fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9128040650 con 0x7f9130071cb0 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: dump: 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912802ce90 con 0x7f9130071cb0 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 5 monclient: authenticate success, global_id 4278 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 10 monclient: _renew_subs 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91301423e0 con 0x7f9130071cb0 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280409f0 con 0x7f9130071cb0 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912d7fa640 10 monclient: dump: 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.188 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f912e7fc640 4 set_mon_vals no callback set 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 10 monclient: _renew_subs 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91301da080 con 0x7f9130071cb0 2024-03-30T07:16:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.185+0000 7f9136b4e640 1 librados: init done 2024-03-30T07:16:51.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.186+0000 7f9136b4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:51.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.186+0000 7f9136b4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:51.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.186+0000 7f9136b4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.186+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913004e030 con 0x7f9130071cb0 2024-03-30T07:16:51.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.456+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9128040b50 con 0x7f9130071cb0 2024-03-30T07:16:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.456+0000 7f912d7fa640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 0x7f911006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.457+0000 7f912ffff640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 0x7f911006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.457+0000 7f912ffff640 10 monclient: get_auth_request con 0x7f911006c5e0 auth_method 0 2024-03-30T07:16:51.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.457+0000 7f912ffff640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 0x7f911006eaa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f91300b6ab0 tx=0x7f9120007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.457+0000 7f912d7fa640 10 client.4278.objecter ms_handle_connect 0x7f911006c5e0 2024-03-30T07:16:51.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.468+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f912807b1c0 con 0x7f9130071cb0 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.468+0000 7f912d7fa640 10 client.4278.objecter ms_dispatch 0x7f9130000b90 osd_map(26..26 src has 1..26) v4 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.468+0000 7f912d7fa640 3 client.4278.objecter handle_osd_map got epochs [26,26] > 0 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.468+0000 7f912d7fa640 3 client.4278.objecter handle_osd_map decoding full epoch 26 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.468+0000 7f912d7fa640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.469+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f9128077500 con 0x7f9130071cb0 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.469+0000 7f912d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:51.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.469+0000 7f912d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:51.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.657+0000 7f9136b4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:51.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.657+0000 7f9136b4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:51.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.657+0000 7f9136b4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.657+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9130069f00 con 0x7f9130071cb0 2024-03-30T07:16:51.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.693+0000 7f912d7fa640 1 -- 172.21.15.47:0/606335432 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+14159 (secure 0 0 0) 0x7f9128039090 con 0x7f9130071cb0 2024-03-30T07:16:51.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.693+0000 7f912d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:16:51.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.693+0000 7f912d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:51.694 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:51.694 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":26,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:16:50.371801+0000","last_up_change":"2024-03-30T07:16:44.160131+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T07:16:32.182687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T07:16:47.206094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":35205},{"type":"v1","addr":"172.21.15.47:6801","nonce":35205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":35205},{"type":"v1","addr":"172.21.15.47:6803","nonce":35205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":35205},{"type":"v1","addr":"172.21.15.47:6807","nonce":35205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":35205},{"type":"v1","addr":"172.21.15.47:6805","nonce":35205}]},"public_addr":"172.21.15.47:6801/35205","cluster_addr":"172.21.15.47:6803/35205","heartbeat_back_addr":"172.21.15.47:6807/35205","heartbeat_front_addr":"172.21.15.47:6805/35205","state":["exists","up"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":35204},{"type":"v1","addr":"172.21.15.47:6809","nonce":35204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":35204},{"type":"v1","addr":"172.21.15.47:6813","nonce":35204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":35204},{"type":"v1","addr":"172.21.15.47:6820","nonce":35204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":35204},{"type":"v1","addr":"172.21.15.47:6816","nonce":35204}]},"public_addr":"172.21.15.47:6809/35204","cluster_addr":"172.21.15.47:6813/35204","heartbeat_back_addr":"172.21.15.47:6820/35204","heartbeat_front_addr":"172.21.15.47:6816/35204","state":["exists","up"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":35207},{"type":"v1","addr":"172.21.15.47:6825","nonce":35207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":35207},{"type":"v1","addr":"172.21.15.47:6827","nonce":35207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":35207},{"type":"v1","addr":"172.21.15.47:6831","nonce":35207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":35207},{"type":"v1","addr":"172.21.15.47:6829","nonce":35207}]},"public_addr":"172.21.15.47:6825/35207","cluster_addr":"172.21.15.47:6827/35207","heartbeat_back_addr":"172.21.15.47:6831/35207","heartbeat_front_addr":"172.21.15.47:6829/35207","state":["exists","up"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":35313},{"type":"v1","addr":"172.21.15.52:6819","nonce":35313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":35313},{"type":"v1","addr":"172.21.15.52:6821","nonce":35313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":35313},{"type":"v1","addr":"172.21.15.52:6825","nonce":35313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":35313},{"type":"v1","addr":"172.21.15.52:6823","nonce":35313}]},"public_addr":"172.21.15.52:6819/35313","cluster_addr":"172.21.15.52:6821/35313","heartbeat_back_addr":"172.21.15.52:6825/35313","heartbeat_front_addr":"172.21.15.52:6823/35313","state":["exists","up"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":35314},{"type":"v1","addr":"172.21.15.52:6829","nonce":35314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6829/35314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":35315},{"type":"v1","addr":"172.21.15.52:6807","nonce":35315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6807/35315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":35316},{"type":"v1","addr":"172.21.15.52:6806","nonce":35316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":35316},{"type":"v1","addr":"172.21.15.52:6812","nonce":35316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":35316},{"type":"v1","addr":"172.21.15.52:6817","nonce":35316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":35316},{"type":"v1","addr":"172.21.15.52:6815","nonce":35316}]},"public_addr":"172.21.15.52:6806/35316","cluster_addr":"172.21.15.52:6812/35316","heartbeat_back_addr":"172.21.15.52:6817/35316","heartbeat_front_addr":"172.21.15.52:6815/35316","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.613520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.651813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.749978+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.080129+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.393136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.240245+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.041599+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:27.954523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:16:51.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 10 librados: watch_flush enter 2024-03-30T07:16:51.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 10 librados: watch_flush exit 2024-03-30T07:16:51.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-30T07:16:51.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 10 client.4278.objecter successfully canceled tick 2024-03-30T07:16:51.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 msgr2=0x7f911006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 0x7f911006eaa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f91300b6ab0 tx=0x7f9120007d30 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 10 monclient: shutdown 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 msgr2=0x7f91301415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9128037040 tx=0x7f9128004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 shutdown_connections 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f911006c5e0 0x7f911006eaa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f9130144a80 0x7f913013f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f913013ec10 0x7f91300aad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 --2- 172.21.15.47:0/606335432 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9130071cb0 0x7f91301415c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 >> 172.21.15.47:0/606335432 conn(0x7f91300b73d0 msgr2=0x7f913013cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 shutdown_connections 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 -- 172.21.15.47:0/606335432 wait complete. 2024-03-30T07:16:51.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.695+0000 7f9136b4e640 1 librados: shutdown 2024-03-30T07:16:51.707 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-30T07:16:51.708 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-30T07:16:51.708 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-30T07:16:51.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-30T07:16:51.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-30T07:16:51.710 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-30T07:16:51.710 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-30T07:16:51.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6a85c41640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6c96dc4640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6c96dc4640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6c96dc4640 10 monclient: monmap: 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.890854+0000 2024-03-30T07:16:51.892 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.890854+0000 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.893 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6a85c41640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.890+0000 7f6a85c41640 10 monclient: monmap: 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.890926+0000 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.890926+0000 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.894 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6a85c41640 1 Processor -- start 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6a85c41640 1 -- start start 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6a85c41640 10 monclient: init 2024-03-30T07:16:51.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6c96dc4640 1 Processor -- start 2024-03-30T07:16:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-30T07:16:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a8006d650 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:51.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6a801431b0 0x7f6a8006da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6c96dc4640 1 -- start start 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.891+0000 7f6c96dc4640 10 monclient: init 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c9006d830 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6c9006dc10 0x7f6c900ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9006dc10 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c90147df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9006ed00 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900af100 con 0x7f6c9006d830 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9013d9a0 con 0x7f6c9006dc10 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90070040 con 0x7f6c9006ed00 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6c96dc4640 10 monclient(hunting): authenticate will time out at 1538.008065s 2024-03-30T07:16:51.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-c con 0x7f6a801431b0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a80149610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-b con 0x7f6a801491c0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a80071c80 con 0x7f6a8006d650 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8006ffa0 con 0x7f6a801431b0 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8006e820 con 0x7f6a801491c0 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6a85c41640 10 monclient(hunting): authenticate will time out at 1538.007693s 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6a801431b0 0x7f6a8006da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6a801431b0 0x7f6a8006da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:43978/0 (socket says 172.21.15.47:43978) 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 1 -- 172.21.15.47:0/2071280850 learned_addr learned my addr 172.21.15.47:0/2071280850 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 10 monclient(hunting): get_auth_request con 0x7f6a801431b0 auth_method 0 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.893+0000 7f6a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6539c34640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6539c34640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.892+0000 7f6539c34640 10 monclient: monmap: 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.893645+0000 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.893645+0000 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.901 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 1 Processor -- start 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 1 -- start start 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 10 monclient: init 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f653406d770 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65340707d0 0x7f653406db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.894+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-a con 0x7f65340707d0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f653413fc90 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653406fca0 con 0x7f653413fc90 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f653406e510 con 0x7f653406d770 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f653406fac0 con 0x7f65340707d0 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6539c34640 10 monclient(hunting): authenticate will time out at 1538.010160s 2024-03-30T07:16:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6a7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6a7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6a7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c94b60640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6c9006d830 0x7f6c9013aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56216/0 (socket says 172.21.15.47:56216) 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c94b60640 1 -- 172.21.15.47:0/1512476785 learned_addr learned my addr 172.21.15.47:0/1512476785 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c8ffff640 1 --2- 172.21.15.47:0/1512476785 >> v2:172.21.15.52:3300/0 conn(0x7f6c9006dc10 0x7f6c900ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c8ffff640 10 monclient(hunting): get_auth_request con 0x7f6c9006dc10 auth_method 0 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.895+0000 7f6c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.896+0000 7f6c8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.896+0000 7f6c8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.896+0000 7f6c8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.897+0000 7f6533fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65340707d0 0x7f653406db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f65340707d0 0x7f653406db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50916/0 (socket says 172.21.15.47:50916) 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 1 -- 172.21.15.47:0/3904609034 learned_addr learned my addr 172.21.15.47:0/3904609034 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 10 monclient(hunting): get_auth_request con 0x7f65340707d0 auth_method 0 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.898+0000 7f6533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 1 -- 172.21.15.47:0/1512476785 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 msgr2=0x7f6c90147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 1 --2- 172.21.15.47:0/1512476785 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c90147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 1 -- 172.21.15.47:0/1512476785 >> v1:172.21.15.52:6789/0 conn(0x7f6c9006d830 legacy=0x7f6c9013aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 1 monclient: found mon.noname-c 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.899+0000 7f6c8ffff640 1 -- 172.21.15.47:0/1512476785 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9013db90 con 0x7f6c9006dc10 2024-03-30T07:16:51.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-03-30T07:16:51.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 1 -- 172.21.15.47:0/2071280850 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 msgr2=0x7f6a80149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 1 --2- 172.21.15.47:0/2071280850 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a80149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 1 -- 172.21.15.47:0/2071280850 >> v1:172.21.15.52:6789/0 conn(0x7f6a8006d650 legacy=0x7f6a80142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 1 monclient: found mon.noname-c 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.903+0000 7f6a7effd640 1 -- 172.21.15.47:0/2071280850 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8006eb00 con 0x7f6a801431b0 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.905+0000 7f6bfdf47640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.905+0000 7f6bfdf47640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.905+0000 7f6bfdf47640 10 monclient: monmap: 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.906070+0000 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.906070+0000 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.910 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.905+0000 7f6c8ffff640 1 --2- 172.21.15.47:0/1512476785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006dc10 0x7f6c900ad9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000a940 tx=0x7f6c800300c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28eab559eea62749 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.906+0000 7f6bfdf47640 1 Processor -- start 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.906+0000 7f6bfdf47640 1 -- start start 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.906+0000 7f6bfdf47640 10 monclient: init 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bf8065f80 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf80b3650 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6bf8141420 0x7f6bf8141850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf8141420 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf80a8ea0 con 0x7f6bf8065f80 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf8066360 con 0x7f6bf8141420 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf80a8bf0 con 0x7f6bf80b3650 2024-03-30T07:16:51.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.908+0000 7f6bfdf47640 10 monclient(hunting): authenticate will time out at 1538.023787s 2024-03-30T07:16:51.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.909+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.909+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.909+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.909+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54738/0 (socket says 172.21.15.47:54738) 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f6bf7fff640 1 -- 172.21.15.47:0/3872067020 learned_addr learned my addr 172.21.15.47:0/3872067020 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f6bf7fff640 10 monclient(hunting): get_auth_request con 0x7f6bf80b3650 auth_method 0 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f6bf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f6bf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f6bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.909+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f891c41a640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f891c41a640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.910+0000 7f891c41a640 10 monclient: monmap: 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.911423+0000 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.911423+0000 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.916 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f891c41a640 1 Processor -- start 2024-03-30T07:16:51.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f891c41a640 1 -- start start 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.911+0000 7f891c41a640 10 monclient: init 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6a7effd640 1 --2- 172.21.15.47:0/2071280850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a801431b0 0x7f6a8006da30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009e50 tx=0x7f6a7802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b20195ca443814 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 1 -- 172.21.15.47:0/1512476785 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80030e70 con 0x7f6c9006dc10 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 10 monclient: dump: 2024-03-30T07:16:51.918 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c96dc4640 5 monclient: authenticate success, global_id 4291 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.912+0000 7f6c96dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 1 -- 172.21.15.47:0/1512476785 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80038440 con 0x7f6c9006dc10 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 4 set_mon_vals no callback set 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 1 -- 172.21.15.47:0/1512476785 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c800387d0 con 0x7f6c9006dc10 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c8f7fe640 10 monclient: dump: 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.921 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.922 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.922 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f890c035070 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f890c02b2b0 0x7f890c0c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-a con 0x7f890c02b2b0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:51.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f890c0c1680 0x7f890c0c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-c con 0x7f890c0c1680 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c02b7c0 con 0x7f890c035070 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 10 monclient: shutdown 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 1 -- 172.21.15.47:0/1512476785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006dc10 msgr2=0x7f6c900ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/1512476785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006dc10 0x7f6c900ad9e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c8000a940 tx=0x7f6c800300c0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c02bc50 con 0x7f890c02b2b0 2024-03-30T07:16:51.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.913+0000 7f891c41a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f891c41a640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c035450 con 0x7f890c0c1680 2024-03-30T07:16:51.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f891c41a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f891c41a640 10 monclient(hunting): authenticate will time out at 1538.029315s 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6bfd746640 1 --2- 172.21.15.47:0/3872067020 >> v2:172.21.15.52:3300/0 conn(0x7f6bf8141420 0x7f6bf8141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 -- 172.21.15.47:0/1512476785 shutdown_connections 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/1512476785 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006dc10 0x7f6c900ad9e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/1512476785 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c90147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 -- 172.21.15.47:0/1512476785 >> 172.21.15.47:0/1512476785 conn(0x7f6c900b2cb0 msgr2=0x7f6c900b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 -- 172.21.15.47:0/1512476785 shutdown_connections 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.914+0000 7f6c96dc4640 1 -- 172.21.15.47:0/1512476785 wait complete. 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bfd746640 10 monclient(hunting): get_auth_request con 0x7f6bf8141420 auth_method 0 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bfd746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bfd746640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bfd746640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6c96dc4640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6c96dc4640 10 monclient: monmap: 2024-03-30T07:16:51.929 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-03-30T07:16:51.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6c96dc4640 1 librados: starting msgr at 2024-03-30T07:16:51.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6c96dc4640 1 librados: starting objecter 2024-03-30T07:16:51.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6c96dc4640 1 Processor -- start 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 1 -- 172.21.15.47:0/3872067020 >> v2:172.21.15.52:3300/0 conn(0x7f6bf8141420 msgr2=0x7f6bf8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 1 --2- 172.21.15.47:0/3872067020 >> v2:172.21.15.52:3300/0 conn(0x7f6bf8141420 0x7f6bf8141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 1 -- 172.21.15.47:0/3872067020 >> v1:172.21.15.47:6789/0 conn(0x7f6bf8065f80 legacy=0x7f6bf813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 1 monclient: found mon.noname-b 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.915+0000 7f6bf7fff640 1 -- 172.21.15.47:0/3872067020 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf80a9330 con 0x7f6bf80b3650 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f891a9b7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f890c0c1680 0x7f890c0c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 1 -- 172.21.15.47:0/3904609034 >> v1:172.21.15.52:6789/0 conn(0x7f653406d770 legacy=0x7f65340700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 1 -- 172.21.15.47:0/3904609034 >> v1:172.21.15.47:6789/0 conn(0x7f653413fc90 legacy=0x7f65341400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 1 monclient: found mon.noname-a 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6533fff640 1 -- 172.21.15.47:0/3904609034 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653406ea00 con 0x7f65340707d0 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6c96dc4640 1 -- start start 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6c96dc4640 1 librados: setting wanted keys 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6c96dc4640 1 librados: calling monclient init 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.916+0000 7f6c96dc4640 10 monclient: init 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006d830 0x7f6c900a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9006d830 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c9006dc10 0x7f6c901db300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9006dc10 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c901db840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9006ed00 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901dae70 con 0x7f6c9006dc10 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900a85e0 con 0x7f6c9006d830 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901dbd80 con 0x7f6c9006ed00 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.917+0000 7f6c96dc4640 10 monclient(hunting): authenticate will time out at 1538.032357s 2024-03-30T07:16:51.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 1 -- 172.21.15.47:0/2071280850 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7802ca30 con 0x7f6a801431b0 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: dump: 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.936 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a85c41640 5 monclient: authenticate success, global_id 4288 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a85c41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 1 -- 172.21.15.47:0/2071280850 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7802cb90 con 0x7f6a801431b0 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a85c41640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a85c41640 4 set_mon_vals no callback set 2024-03-30T07:16:51.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f890c0c1680 0x7f890c0c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44006/0 (socket says 172.21.15.47:44006) 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 1 -- 172.21.15.47:0/2530302991 learned_addr learned my addr 172.21.15.47:0/2530302991 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 10 monclient(hunting): get_auth_request con 0x7f890c0c1680 auth_method 0 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f891a9b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 1 -- 172.21.15.47:0/2071280850 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a780343f0 con 0x7f6a801431b0 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:51.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6a7e7fc640 10 monclient: dump: 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.918+0000 7f6bf7fff640 1 --2- 172.21.15.47:0/3872067020 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009870 tx=0x7f6bec02b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48d869d94cb9f50b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 10 monclient: shutdown 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 -- 172.21.15.47:0/2071280850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a801431b0 msgr2=0x7f6a8006da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 --2- 172.21.15.47:0/2071280850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a801431b0 0x7f6a8006da30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009e50 tx=0x7f6a7802be90 comp rx=0 tx=0).stop 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 -- 172.21.15.47:0/2071280850 shutdown_connections 2024-03-30T07:16:51.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 --2- 172.21.15.47:0/2071280850 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a801431b0 0x7f6a8006da30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 --2- 172.21.15.47:0/2071280850 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a80149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 -- 172.21.15.47:0/2071280850 >> 172.21.15.47:0/2071280850 conn(0x7f6a800b2cb0 msgr2=0x7f6a800b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 -- 172.21.15.47:0/2071280850 shutdown_connections 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 1 -- 172.21.15.47:0/2071280850 wait complete. 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.919+0000 7f6a85c41640 10 monclient: monmap: 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:51.943 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 librados: starting msgr at 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 librados: starting objecter 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 Processor -- start 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 -- start start 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 librados: setting wanted keys 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 1 librados: calling monclient init 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.920+0000 7f6a85c41640 10 monclient: init 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a8006d650 0x7f6a800a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6533fff640 1 --2- 172.21.15.47:0/3904609034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65340707d0 0x7f653406db50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65240091e0 tx=0x7f652402ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a826335127500215 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-b con 0x7f6a8006d650 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:51.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a801431b0 0x7f6a8013e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-a con 0x7f6a801431b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a8013e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): picked mon.noname-c con 0x7f6a801491c0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800a9f10 con 0x7f6a801431b0 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a800aa1a0 con 0x7f6a8006d650 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80140020 con 0x7f6a801491c0 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7f6a85c41640 10 monclient(hunting): authenticate will time out at 1538.036469s 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7fbd8567d640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7fbd8567d640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.921+0000 7fbd8567d640 10 monclient: monmap: 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.922696+0000 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.922696+0000 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.947 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.922+0000 7fbd8567d640 1 Processor -- start 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.922+0000 7fbd8567d640 1 -- start start 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.922+0000 7fbd8567d640 10 monclient: init 2024-03-30T07:16:51.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.923+0000 7f6c95361640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c901db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.923+0000 7f6a7ffff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a8013e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.923+0000 7fbd8567d640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.923+0000 7fbd8567d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-30T07:16:51.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.923+0000 7fbd8567d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-b con 0x7fbd8006fa60 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd8013c840 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fbd8013fb40 0x7fbd80149c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-c con 0x7fbd8013fb40 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8013cca0 con 0x7fbd8013c840 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd800719d0 con 0x7fbd8013fb40 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80071cb0 con 0x7fbd8006fa60 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7fbd8567d640 10 monclient(hunting): authenticate will time out at 1538.039268s 2024-03-30T07:16:51.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 1 -- 172.21.15.47:0/2530302991 >> v2:172.21.15.47:3300/0 conn(0x7f890c02b2b0 msgr2=0x7f890c0c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 1 --2- 172.21.15.47:0/2530302991 >> v2:172.21.15.47:3300/0 conn(0x7f890c02b2b0 0x7f890c0c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 1 -- 172.21.15.47:0/2530302991 >> v1:172.21.15.47:6789/0 conn(0x7f890c035070 legacy=0x7f890c0c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 1 monclient: found mon.noname-c 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f891a9b7640 1 -- 172.21.15.47:0/2530302991 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c02c980 con 0x7f890c0c1680 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c901db840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54754/0 (socket says 172.21.15.47:54754) 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 1 -- 172.21.15.47:0/2524618362 learned_addr learned my addr 172.21.15.47:0/2524618362 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 10 monclient(hunting): get_auth_request con 0x7f6c9006ed00 auth_method 0 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6c95361640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a8013e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54760/0 (socket says 172.21.15.47:54760) 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 1 -- 172.21.15.47:0/4198320581 learned_addr learned my addr 172.21.15.47:0/4198320581 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 10 monclient(hunting): get_auth_request con 0x7f6a801491c0 auth_method 0 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.924+0000 7f6a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7ff52ad7e640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7ff52ad7e640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7ff52ad7e640 10 monclient: monmap: 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.925870+0000 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.925870+0000 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.925+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7f6c95361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7f6c95361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7f6c95361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7ff52ad7e640 1 Processor -- start 2024-03-30T07:16:51.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7f6c94b60640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006d830 0x7f6c900a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7ff52ad7e640 1 -- start start 2024-03-30T07:16:51.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.926+0000 7ff52ad7e640 10 monclient: init 2024-03-30T07:16:51.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): get_auth_request con 0x7f6c9006d830 auth_method 0 2024-03-30T07:16:51.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6a7f7fe640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a8006d650 0x7f6a800a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7f6c94b60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff52406e650 0x7ff52406ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff52406e650 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff52406d9b0 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:51.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.927+0000 7ff52ad7e640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff5240707e0 0x7ff524070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7f6a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6a8006d650 auth_method 0 2024-03-30T07:16:51.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7f6a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7f6a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-a con 0x7ff5240707e0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5240701c0 con 0x7ff52406d9b0 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52406f5a0 con 0x7ff5240707e0 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5240ad9a0 con 0x7ff52406e650 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7ff52ad7e640 10 monclient(hunting): authenticate will time out at 1538.043271s 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.928+0000 7f6a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.929+0000 7f6a7ffff640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.929+0000 7f6a7ffff640 1 -- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a8006d650 msgr2=0x7f6a800a99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.929+0000 7f6a7ffff640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6a8006d650 0x7f6a800a99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.929+0000 7f6a7ffff640 1 -- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a801431b0 msgr2=0x7f6a8013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6a801431b0 0x7f6a8013e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 1 monclient: found mon.noname-c 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6a7ffff640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78009b30 con 0x7f6a801491c0 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 -- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 msgr2=0x7f6c901db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c9006ed00 0x7f6c901db840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 -- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c9006dc10 msgr2=0x7f6c901db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c9006dc10 0x7f6c901db300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 monclient: found mon.noname-b 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.930+0000 7f6c94b60640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8000a5a0 con 0x7f6c9006d830 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.933+0000 7ff52931b640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff5240707e0 0x7ff524070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff5240707e0 0x7ff524070bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50946/0 (socket says 172.21.15.47:50946) 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 1 -- 172.21.15.47:0/2699719488 learned_addr learned my addr 172.21.15.47:0/2699719488 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): get_auth_request con 0x7ff5240707e0 auth_method 0 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.934+0000 7ff52931b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 1 -- 172.21.15.47:0/2699719488 >> v2:172.21.15.52:3300/0 conn(0x7ff52406e650 msgr2=0x7ff52406ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 1 --2- 172.21.15.47:0/2699719488 >> v2:172.21.15.52:3300/0 conn(0x7ff52406e650 0x7ff52406ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 1 -- 172.21.15.47:0/2699719488 >> v1:172.21.15.52:6789/0 conn(0x7ff52406d9b0 legacy=0x7ff52413ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 1 monclient: found mon.noname-a 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.936+0000 7ff52931b640 1 -- 172.21.15.47:0/2699719488 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52413b540 con 0x7ff5240707e0 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.937+0000 7f6c94b60640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006d830 0x7f6c900a9f00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c740 tx=0x7f6c8800cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.937+0000 7f6c8d7fa640 10 client.?.objecter ms_handle_connect 0x7f6c9006d830 2024-03-30T07:16:51.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.937+0000 7f6c8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.939+0000 7fbd7f7fe640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fbd8013fb40 0x7fbd80149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7ff5af451640 10 monclient: get_monmap_and_config 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7ff5af451640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7ff5af451640 10 monclient: monmap: 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.967 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:51.940780+0000 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:51.940780+0000 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6a7ffff640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a8013e7e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400b440 tx=0x7f6a7400b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6a67fff640 10 client.?.objecter ms_handle_connect 0x7f6a801491c0 2024-03-30T07:16:51.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6a67fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:51.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 1 -- 172.21.15.47:0/3872067020 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec033620 con 0x7f6bf80b3650 2024-03-30T07:16:51.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:51.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7ff52931b640 1 --2- 172.21.15.47:0/2699719488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5240707e0 0x7ff524070bc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff5240b1070 tx=0x7ff51c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ff6a4d6d3abe32 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: dump: 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.970 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 1 -- 172.21.15.47:0/3872067020 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec033780 con 0x7f6bf80b3650 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 1 -- 172.21.15.47:0/3872067020 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec033b10 con 0x7f6bf80b3650 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bf77fe640 10 monclient: dump: 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.971 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bfdf47640 5 monclient: authenticate success, global_id 4229 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bfdf47640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7f6bfdf47640 4 set_mon_vals no callback set 2024-03-30T07:16:51.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.940+0000 7fbd7effd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7ff5af451640 1 Processor -- start 2024-03-30T07:16:51.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 10 monclient: shutdown 2024-03-30T07:16:51.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:51.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3872067020 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 msgr2=0x7f6bf80a86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7ff5af451640 1 -- start start 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7ff5af451640 10 monclient: init 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3872067020 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bec009870 tx=0x7f6bec02b7d0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3872067020 shutdown_connections 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3872067020 >> v2:172.21.15.52:3300/0 conn(0x7f6bf8141420 0x7f6bf8141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3872067020 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf80b3650 0x7f6bf80a86b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3872067020 >> 172.21.15.47:0/3872067020 conn(0x7f6bf80b7120 msgr2=0x7f6bf80b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3872067020 shutdown_connections 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54774/0 (socket says 172.21.15.47:54774) 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 1 -- 172.21.15.47:0/3358952801 learned_addr learned my addr 172.21.15.47:0/3358952801 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 10 monclient(hunting): get_auth_request con 0x7fbd8006fa60 auth_method 0 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7fbd7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.941+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3872067020 wait complete. 2024-03-30T07:16:51.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-30T07:16:51.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbd8013fb40 auth_method 0 2024-03-30T07:16:51.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7ff5a80b0120 0x7ff5a80a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a80b0120 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5a8144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5a80b0d20 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a80a8bd0 con 0x7ff5a80b0d20 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a8144f10 con 0x7ff5a8144ab0 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a80aa4f0 con 0x7ff5a80b0120 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7ff5af451640 10 monclient(hunting): authenticate will time out at 1538.057576s 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7f6bfdf47640 10 monclient: build_initial_monmap 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7f6bfdf47640 10 monclient: monmap: 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.979 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.942+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 librados: starting msgr at 2024-03-30T07:16:51.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 librados: starting objecter 2024-03-30T07:16:51.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 Processor -- start 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 -- start start 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 librados: setting wanted keys 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 1 librados: calling monclient init 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.943+0000 7f6bfdf47640 10 monclient: init 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bf8065f80 0x7f6bf81374b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf8065f80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf8141420 0x7f6bf81d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf8141420 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81da1f0 0x7f6bf80b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf81da1f0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf813e190 con 0x7f6bf81da1f0 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81379f0 con 0x7f6bf8065f80 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf81d9540 con 0x7f6bf8141420 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f6bfdf47640 10 monclient(hunting): authenticate will time out at 1538.059581s 2024-03-30T07:16:51.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.944+0000 7f891a9b7640 1 --2- 172.21.15.47:0/2530302991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c0c1680 0x7f890c0c41b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8914074490 tx=0x7f89140a66f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d20710b253b065c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 1 -- 172.21.15.47:0/3358952801 >> v2:172.21.15.52:3300/0 conn(0x7fbd8013fb40 msgr2=0x7fbd80149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 1 --2- 172.21.15.47:0/3358952801 >> v2:172.21.15.52:3300/0 conn(0x7fbd8013fb40 0x7fbd80149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 1 -- 172.21.15.47:0/3358952801 >> v1:172.21.15.47:6789/0 conn(0x7fbd8013c840 legacy=0x7fbd8013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 1 monclient: found mon.noname-b 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.945+0000 7fbd7effd640 1 -- 172.21.15.47:0/3358952801 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80070800 con 0x7fbd8006fa60 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7ff5ac9ec640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7ff5a8144ab0 0x7ff5a80712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56246/0 (socket says 172.21.15.47:56246) 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7ff5ac9ec640 1 -- 172.21.15.47:0/2733356548 learned_addr learned my addr 172.21.15.47:0/2733356548 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74011830 con 0x7f6a801491c0 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 10 monclient: dump: 2024-03-30T07:16:51.986 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 5 monclient: authenticate success, global_id 4232 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 10 monclient: _renew_subs 2024-03-30T07:16:51.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a800a9310 con 0x7f6a801491c0 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 10 monclient: _renew_subs 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8013ef00 con 0x7f6a801491c0 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.948+0000 7f6a85c41640 1 librados: init done 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 10 client.4232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 15 client.4232.objecter _session_command_op_assign -1 1 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 20 client.4232.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:51.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a801501b0 con 0x7f6a801491c0 2024-03-30T07:16:51.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800aa50 con 0x7f6c9006d830 2024-03-30T07:16:51.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:51.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: dump: 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 5 monclient: authenticate success, global_id 4300 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 10 monclient: _renew_subs 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9013ff00 con 0x7f6c9006d830 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800abb0 con 0x7f6c9006d830 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800ce50 con 0x7f6c9006d830 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 10 monclient: _renew_subs 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c900a90e0 con 0x7f6c9006d830 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c96dc4640 1 librados: init done 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8effd640 4 set_mon_vals no callback set 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:51.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.949+0000 7f6c8d7fa640 10 monclient: dump: 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 15 client.4300.objecter _session_command_op_assign -1 1 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 20 client.4300.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:51.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:51.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.950+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c900b37d0 con 0x7f6c9006d830 2024-03-30T07:16:51.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.951+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a74011990 con 0x7f6a801491c0 2024-03-30T07:16:51.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a7dffb640 4 set_mon_vals no callback set 2024-03-30T07:16:51.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf8141420 0x7f6bf81d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74011830 con 0x7f6a801491c0 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6a67fff640 10 monclient: dump: 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.994 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf8141420 0x7f6bf81d9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54778/0 (socket says 172.21.15.47:54778) 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 1 -- 172.21.15.47:0/3643531402 learned_addr learned my addr 172.21.15.47:0/3643531402 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 10 monclient(hunting): get_auth_request con 0x7f6bf8141420 auth_method 0 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.952+0000 7f6bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.953+0000 7ff5ad1ed640 1 --2- 172.21.15.47:0/2733356548 >> v2:172.21.15.52:3300/0 conn(0x7ff5a80b0120 0x7ff5a80a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:51.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:51.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7f6bf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7ff5ad1ed640 10 monclient(hunting): get_auth_request con 0x7ff5a80b0120 auth_method 0 2024-03-30T07:16:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7ff5ad1ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7ff5ad1ed640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7ff5ad1ed640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.954+0000 7f6bfcf45640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bf8065f80 0x7f6bf81374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 1 -- 172.21.15.47:0/2699719488 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c02c9f0 con 0x7ff5240707e0 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: dump: 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:51.998 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 1 -- 172.21.15.47:0/2699719488 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51c02cb50 con 0x7ff5240707e0 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 1 -- 172.21.15.47:0/2699719488 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c0343f0 con 0x7ff5240707e0 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff52ad7e640 5 monclient: authenticate success, global_id 4284 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff52ad7e640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff52ad7e640 4 set_mon_vals no callback set 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.955+0000 7ff5237fe640 10 monclient: dump: 2024-03-30T07:16:51.999 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.000 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff5ad1ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff5ad1ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff5ad1ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 10 monclient: shutdown 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 -- 172.21.15.47:0/2699719488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5240707e0 msgr2=0x7ff524070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/2699719488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5240707e0 0x7ff524070bc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff5240b1070 tx=0x7ff51c02be50 comp rx=0 tx=0).stop 2024-03-30T07:16:52.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7f6bfcf45640 10 monclient(hunting): get_auth_request con 0x7f6bf8065f80 auth_method 0 2024-03-30T07:16:52.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7f6bfcf45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7f6bfcf45640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7f6bfcf45640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 -- 172.21.15.47:0/2699719488 shutdown_connections 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/2699719488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5240707e0 0x7ff524070bc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/2699719488 >> v2:172.21.15.52:3300/0 conn(0x7ff52406e650 0x7ff52406ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 -- 172.21.15.47:0/2699719488 >> 172.21.15.47:0/2699719488 conn(0x7ff5240b3f70 msgr2=0x7ff524069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 -- 172.21.15.47:0/2699719488 shutdown_connections 2024-03-30T07:16:52.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.956+0000 7ff52ad7e640 1 -- 172.21.15.47:0/2699719488 wait complete. 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 1 -- 172.21.15.47:0/2530302991 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140ab830 con 0x7f890c0c1680 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: dump: 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.004 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 1 -- 172.21.15.47:0/2530302991 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89140a9050 con 0x7f890c0c1680 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 1 -- 172.21.15.47:0/2530302991 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140abc50 con 0x7f890c0c1680 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f89191b4640 10 monclient: dump: 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.005 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f891c41a640 5 monclient: authenticate success, global_id 4297 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f891c41a640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7f891c41a640 4 set_mon_vals no callback set 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7ff52ad7e640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7ff52ad7e640 10 monclient: monmap: 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.007 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7ff52ad7e640 1 librados: starting msgr at 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7ff52ad7e640 1 librados: starting objecter 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.957+0000 7ff52ad7e640 1 Processor -- start 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 10 monclient: shutdown 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 -- 172.21.15.47:0/2530302991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c0c1680 msgr2=0x7f890c0c41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 --2- 172.21.15.47:0/2530302991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c0c1680 0x7f890c0c41b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8914074490 tx=0x7f89140a66f0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff52ad7e640 1 -- start start 2024-03-30T07:16:52.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff52ad7e640 1 librados: setting wanted keys 2024-03-30T07:16:52.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff52ad7e640 1 librados: calling monclient init 2024-03-30T07:16:52.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff52ad7e640 10 monclient: init 2024-03-30T07:16:52.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 -- 172.21.15.47:0/2530302991 shutdown_connections 2024-03-30T07:16:52.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 --2- 172.21.15.47:0/2530302991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c0c1680 0x7f890c0c41b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 --2- 172.21.15.47:0/2530302991 >> v2:172.21.15.47:3300/0 conn(0x7f890c02b2b0 0x7f890c0c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 -- 172.21.15.47:0/2530302991 >> 172.21.15.47:0/2530302991 conn(0x7f890c006510 msgr2=0x7f890c006920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-03-30T07:16:52.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7f891c41a640 1 -- 172.21.15.47:0/2530302991 shutdown_connections 2024-03-30T07:16:52.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 1 -- 172.21.15.47:0/2733356548 >> v1:172.21.15.52:6789/0 conn(0x7ff5a8144ab0 legacy=0x7ff5a80712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 1 -- 172.21.15.47:0/2733356548 >> v1:172.21.15.47:6789/0 conn(0x7ff5a80b0d20 legacy=0x7ff5a814be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff5ad1ed640 1 -- 172.21.15.47:0/2733356548 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80b3780 con 0x7ff5a80b0120 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.958+0000 7ff52ad7e640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 0x7ff524139f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-a con 0x7ff52406d9b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff52406e650 0x7ff524136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-b con 0x7ff52406e650 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5240707e0 0x7ff524137de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff5240707e0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff524135f10 con 0x7ff52406d9b0 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff524138320 con 0x7ff52406e650 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff524135820 con 0x7ff5240707e0 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7ff52ad7e640 10 monclient(hunting): authenticate will time out at 1538.074338s 2024-03-30T07:16:52.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7f891c41a640 1 -- 172.21.15.47:0/2530302991 wait complete. 2024-03-30T07:16:52.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7f6bfcf45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7f6bfcf45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.959+0000 7f6bfcf45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.960+0000 7f891c41a640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.960+0000 7f891c41a640 10 monclient: monmap: 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.016 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 librados: starting msgr at 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 librados: starting objecter 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 Processor -- start 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 -- start start 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 librados: setting wanted keys 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 1 librados: calling monclient init 2024-03-30T07:16:52.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f891c41a640 10 monclient: init 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 1 -- 172.21.15.47:0/3904609034 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652402da80 con 0x7f65340707d0 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: dump: 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.018 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 1 -- 172.21.15.47:0/3904609034 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652402dbe0 con 0x7f65340707d0 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 1 -- 172.21.15.47:0/3904609034 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524035440 con 0x7f65340707d0 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f65337fe640 10 monclient: dump: 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.019 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f6539c34640 5 monclient: authenticate success, global_id 4281 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f6539c34640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.961+0000 7f6539c34640 4 set_mon_vals no callback set 2024-03-30T07:16:52.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfd746640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81da1f0 0x7f6bf80b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:52.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f890c02b2b0 0x7f890c0301d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-c con 0x7f890c02b2b0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c035070 0x7f890c15bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-b con 0x7f890c035070 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 0x7f890c0ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): picked mon.noname-a con 0x7f890c15c9b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0bd720 con 0x7f890c15c9b0 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c15c520 con 0x7f890c035070 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c02fd80 con 0x7f890c02b2b0 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f891c41a640 10 monclient(hunting): authenticate will time out at 1538.077468s 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfd746640 10 monclient(hunting): get_auth_request con 0x7f6bf81da1f0 auth_method 0 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfd746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfd746640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfd746640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 -- 172.21.15.47:0/3643531402 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf8141420 msgr2=0x7f6bf81d9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bf8141420 0x7f6bf81d9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 -- 172.21.15.47:0/3643531402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81da1f0 msgr2=0x7f6bf80b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bf81da1f0 0x7f6bf80b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6bfcf45640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec0094d0 con 0x7f6bf8065f80 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 10 monclient: shutdown 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 -- 172.21.15.47:0/3904609034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65340707d0 msgr2=0x7f653406db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 --2- 172.21.15.47:0/3904609034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65340707d0 0x7f653406db50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f65240091e0 tx=0x7f652402ccd0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 -- 172.21.15.47:0/3904609034 shutdown_connections 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 --2- 172.21.15.47:0/3904609034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f65340707d0 0x7f653406db50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 -- 172.21.15.47:0/3904609034 >> 172.21.15.47:0/3904609034 conn(0x7f65340af610 msgr2=0x7f65340afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 -- 172.21.15.47:0/3904609034 shutdown_connections 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.962+0000 7f6539c34640 1 -- 172.21.15.47:0/3904609034 wait complete. 2024-03-30T07:16:52.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.963+0000 7f6539c34640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.963+0000 7f6539c34640 10 monclient: monmap: 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.963+0000 7f6539c34640 1 librados: starting msgr at 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.963+0000 7f6539c34640 1 librados: starting objecter 2024-03-30T07:16:52.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 Processor -- start 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 -- start start 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 librados: setting wanted keys 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 librados: calling monclient init 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient: init 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f653406d770 0x7f65341d9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-c con 0x7f653406d770 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 0x7f653413def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-b con 0x7f65340707d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f653413fc90 0x7f653413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient(hunting): picked mon.noname-a con 0x7f653413fc90 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653413e970 con 0x7f653413fc90 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.964+0000 7f6539c34640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65341c74e0 con 0x7f65340707d0 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.965+0000 7f6539c34640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.965+0000 7f6539c34640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65340aa0f0 con 0x7f653406d770 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.965+0000 7f6539c34640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.965+0000 7f6539c34640 10 monclient(hunting): authenticate will time out at 1538.080143s 2024-03-30T07:16:52.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.966+0000 7ff5ad1ed640 1 --2- 172.21.15.47:0/2733356548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0120 0x7ff5a80a84a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40097a0 tx=0x7ff5a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8c3bf14f380052 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.967+0000 7ff528b1a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 0x7ff524139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 0x7ff524139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50962/0 (socket says 172.21.15.47:50962) 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 1 -- 172.21.15.47:0/3219478015 learned_addr learned my addr 172.21.15.47:0/3219478015 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): get_auth_request con 0x7ff52406d9b0 auth_method 0 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7ff528b1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.968+0000 7f891a9b7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 0x7f890c0ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 0x7f653413def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 0x7f890c0ba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:50968/0 (socket says 172.21.15.47:50968) 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 1 -- 172.21.15.47:0/728663371 learned_addr learned my addr 172.21.15.47:0/728663371 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 10 monclient(hunting): get_auth_request con 0x7f890c15c9b0 auth_method 0 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f891a9b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 0x7f653413def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44096/0 (socket says 172.21.15.47:44096) 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 1 -- 172.21.15.47:0/2917011029 learned_addr learned my addr 172.21.15.47:0/2917011029 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 10 monclient(hunting): get_auth_request con 0x7f65340707d0 auth_method 0 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.969+0000 7f6533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.970+0000 7f6bfcf45640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bf8065f80 0x7f6bf81374b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c470 tx=0x7f6be800c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.970+0000 7f6bf57fa640 10 client.?.objecter ms_handle_connect 0x7f6bf8065f80 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.970+0000 7f6bf57fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.971+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.971+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.971+0000 7f6533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-03-30T07:16:52.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 -- 172.21.15.47:0/3219478015 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5240707e0 msgr2=0x7ff524137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5240707e0 0x7ff524137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 -- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff52406e650 msgr2=0x7ff524136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff52406e650 0x7ff524136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.972+0000 7ff528b1a640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c009b30 con 0x7ff52406d9b0 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.973+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.973+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.973+0000 7f891a9b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 -- 172.21.15.47:0/2917011029 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f653406d770 msgr2=0x7f65341d9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f653406d770 0x7f65341d9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 -- 172.21.15.47:0/2917011029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f653413fc90 msgr2=0x7f653413e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f653413fc90 0x7f653413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.975+0000 7f6533fff640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524009d60 con 0x7f65340707d0 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 -- 172.21.15.47:0/728663371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f890c02b2b0 msgr2=0x7f890c0301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f890c02b2b0 0x7f890c0301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 -- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c035070 msgr2=0x7f890c15bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f890c035070 0x7f890c15bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.977+0000 7f891a9b7640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891406be70 con 0x7f890c15c9b0 2024-03-30T07:16:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7f6533fff640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 0x7f653413def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6524009300 tx=0x7f65240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7f65317fa640 10 client.?.objecter ms_handle_connect 0x7f65340707d0 2024-03-30T07:16:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7f65317fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7ff528b1a640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 0x7ff524139f20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b4a0 tx=0x7ff51800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7ff5217fa640 10 client.?.objecter ms_handle_connect 0x7ff52406d9b0 2024-03-30T07:16:52.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.978+0000 7ff5217fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.981+0000 7f891a9b7640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 0x7f890c0ba240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8914063b00 tx=0x7f8914068530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.981+0000 7f890affd640 10 client.?.objecter ms_handle_connect 0x7f890c15c9b0 2024-03-30T07:16:52.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.981+0000 7f890affd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.985+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a74011b00 con 0x7f6a801491c0 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.986+0000 7f6a67fff640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6a6c06c5f0 0x7f6a6c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.986+0000 7f6a7f7fe640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6a6c06c5f0 0x7f6a6c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.986+0000 7f6a7f7fe640 10 monclient: get_auth_request con 0x7f6a6c06c5f0 auth_method 0 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7f6a67fff640 10 client.4232.objecter ms_handle_connect 0x7f6a6c06c5f0 2024-03-30T07:16:52.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7f6a7f7fe640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6a6c06c5f0 0x7f6a6c06eab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000f60 tx=0x7f6a70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.986+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518010610 con 0x7ff52406d9b0 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: dump: 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.045 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff518010790 con 0x7ff52406d9b0 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518010b40 con 0x7ff52406d9b0 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff5217fa640 10 monclient: dump: 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.046 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff522ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 5 monclient: authenticate success, global_id 4290 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 10 monclient: _renew_subs 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5241359e0 con 0x7ff52406d9b0 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 10 monclient: _renew_subs 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5241d7ce0 con 0x7ff52406d9b0 2024-03-30T07:16:52.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.987+0000 7ff52ad7e640 1 librados: init done 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 15 client.4290.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 20 client.4290.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.988+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff524071350 con 0x7ff52406d9b0 2024-03-30T07:16:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140b48f0 con 0x7f890c15c9b0 2024-03-30T07:16:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 10 monclient: dump: 2024-03-30T07:16:52.049 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f890affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 5 monclient: authenticate success, global_id 4293 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 10 monclient: _renew_subs 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c15c320 con 0x7f890c15c9b0 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 10 monclient: _renew_subs 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c0bac20 con 0x7f890c15c9b0 2024-03-30T07:16:52.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f891c41a640 1 librados: init done 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80035f0 con 0x7f6bf8065f80 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: dump: 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8003750 con 0x7f6bf8065f80 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 5 monclient: authenticate success, global_id 4315 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 10 monclient: _renew_subs 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf80b66a0 con 0x7f6bf8065f80 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8003b00 con 0x7f6bf8065f80 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf57fa640 10 monclient: dump: 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.053 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 10 monclient: _renew_subs 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf80b63d0 con 0x7f6bf8065f80 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bfdf47640 1 librados: init done 2024-03-30T07:16:52.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.989+0000 7f6bf6ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 15 client.4293.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 20 client.4293.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f890c0baa80 con 0x7f890c15c9b0 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 10 client.4315.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 15 client.4315.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 20 client.4315.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.990+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6bf814bc70 con 0x7f6bf8065f80 2024-03-30T07:16:52.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6a74058c70 con 0x7f6a801491c0 2024-03-30T07:16:52.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 10 client.4232.objecter ms_dispatch 0x7f6a80000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 3 client.4232.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 3 client.4232.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 10 client.4232.objecter checking command 1 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6a6c0729d0 0x7f6a6c074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter _get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 15 client.4232.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter _get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 15 client.4232.objecter _session_command_op_assign 0 1 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 4 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 10 client.4232.objecter _send_command 1 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a6c0754a0 con 0x7f6a6c0729d0 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a740548b0 con 0x7f6a801491c0 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.991+0000 7f6a67fff640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.992+0000 7f6a7effd640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6a6c0729d0 0x7f6a6c074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.993+0000 7f6a7effd640 10 monclient: get_auth_request con 0x7f6a6c0729d0 auth_method 0 2024-03-30T07:16:52.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.993+0000 7fbd7effd640 1 --2- 172.21.15.47:0/3358952801 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009870 tx=0x7fbd7002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77ff3505c7e92911 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.997+0000 7f6a7effd640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6a6c0729d0 0x7f6a6c074df0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:51.997+0000 7f6a67fff640 10 client.4232.objecter ms_handle_connect 0x7f6a6c0729d0 2024-03-30T07:16:52.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.003+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c88012420 con 0x7f6c9006d830 2024-03-30T07:16:52.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.003+0000 7f6c8d7fa640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6c7c06c590 0x7f6c7c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.003+0000 7f6c8ffff640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6c7c06c590 0x7f6c7c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.004+0000 7f6c8ffff640 10 monclient: get_auth_request con 0x7f6c7c06c590 auth_method 0 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.004+0000 7f6c8ffff640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6c7c06c590 0x7f6c7c06ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c8003a000 tx=0x7f6c80032000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.004+0000 7f6c8d7fa640 10 client.4300.objecter ms_handle_connect 0x7f6c7c06c590 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.005+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6c88058ff0 con 0x7f6c9006d830 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.005+0000 7f6c8d7fa640 10 client.4300.objecter ms_dispatch 0x7f6c90000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.005+0000 7f6c8d7fa640 3 client.4300.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.005+0000 7f6c8d7fa640 3 client.4300.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 10 client.4300.objecter checking command 1 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f6c7c072970 0x7f6c7c074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter _get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 15 client.4300.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter _get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 15 client.4300.objecter _session_command_op_assign 3 1 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 4 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 10 client.4300.objecter _send_command 1 2024-03-30T07:16:52.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c7c075440 con 0x7f6c7c072970 2024-03-30T07:16:52.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c8d7fa640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.006+0000 7f6c95361640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f6c7c072970 0x7f6c7c074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7f6c95361640 10 monclient: get_auth_request con 0x7f6c7c072970 auth_method 0 2024-03-30T07:16:52.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 1 -- 172.21.15.47:0/3358952801 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd700317c0 con 0x7fbd8006fa60 2024-03-30T07:16:52.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: dump: 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 1 -- 172.21.15.47:0/3358952801 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd70031920 con 0x7fbd8006fa60 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 1 -- 172.21.15.47:0/3358952801 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd70031cb0 con 0x7fbd8006fa60 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd7e7fc640 10 monclient: dump: 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.068 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd8567d640 5 monclient: authenticate success, global_id 4238 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd8567d640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.007+0000 7fbd8567d640 4 set_mon_vals no callback set 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.008+0000 7fbd8567d640 10 monclient: shutdown 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.008+0000 7fbd8567d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.008+0000 7fbd8567d640 1 -- 172.21.15.47:0/3358952801 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 msgr2=0x7fbd8013c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.008+0000 7fbd8567d640 1 --2- 172.21.15.47:0/3358952801 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009870 tx=0x7fbd7002b860 comp rx=0 tx=0).stop 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 -- 172.21.15.47:0/3358952801 shutdown_connections 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 --2- 172.21.15.47:0/3358952801 >> v2:172.21.15.52:3300/0 conn(0x7fbd8013fb40 0x7fbd80149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 --2- 172.21.15.47:0/3358952801 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8006fa60 0x7fbd8013c300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 -- 172.21.15.47:0/3358952801 >> 172.21.15.47:0/3358952801 conn(0x7fbd800b4e40 msgr2=0x7fbd800b5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 -- 172.21.15.47:0/3358952801 shutdown_connections 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.009+0000 7fbd8567d640 1 -- 172.21.15.47:0/3358952801 wait complete. 2024-03-30T07:16:52.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 10 monclient: monmap: 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.070 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 1 -- 172.21.15.47:0/2733356548 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4031830 con 0x7ff5a80b0120 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: dump: 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 1 -- 172.21.15.47:0/2733356548 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a4031990 con 0x7ff5a80b0120 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 5 monclient: authenticate success, global_id 4312 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 4 set_mon_vals no callback set 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 1 -- 172.21.15.47:0/2733356548 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a4031d20 con 0x7ff5a80b0120 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff59ffff640 10 monclient: dump: 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.074 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 1 librados: starting msgr at 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 1 librados: starting objecter 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 1 Processor -- start 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 10 monclient: shutdown 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 1 -- 172.21.15.47:0/2733356548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0120 msgr2=0x7ff5a80a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7ff5af451640 1 --2- 172.21.15.47:0/2733356548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0120 0x7ff5a80a84a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40097a0 tx=0x7ff5a402b8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.010+0000 7fbd8567d640 1 -- start start 2024-03-30T07:16:52.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7fbd8567d640 1 librados: setting wanted keys 2024-03-30T07:16:52.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7fbd8567d640 1 librados: calling monclient init 2024-03-30T07:16:52.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7fbd8567d640 10 monclient: init 2024-03-30T07:16:52.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7ff5af451640 1 -- 172.21.15.47:0/2733356548 shutdown_connections 2024-03-30T07:16:52.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7ff5af451640 1 --2- 172.21.15.47:0/2733356548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0120 0x7ff5a80a84a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7ff5af451640 1 -- 172.21.15.47:0/2733356548 >> 172.21.15.47:0/2733356548 conn(0x7ff5a80b70b0 msgr2=0x7ff5a80b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7ff5af451640 1 -- 172.21.15.47:0/2733356548 shutdown_connections 2024-03-30T07:16:52.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7ff5af451640 1 -- 172.21.15.47:0/2733356548 wait complete. 2024-03-30T07:16:52.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7f6c95361640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f6c7c072970 0x7f6c7c074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.011+0000 7f6c8d7fa640 10 client.4300.objecter ms_handle_connect 0x7f6c7c072970 2024-03-30T07:16:52.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd8006fa60 0x7fbd80139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-a con 0x7fbd8006fa60 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 0x7fbd8006acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7ff5af451640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7ff5af451640 10 monclient: monmap: 2024-03-30T07:16:52.079 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.080 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-c con 0x7fbd8013c840 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd8013fb40 0x7fbd801d9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): picked mon.noname-b con 0x7fbd8013fb40 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006a8d0 con 0x7fbd8006fa60 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8006b230 con 0x7fbd8013fb40 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd801cafe0 con 0x7fbd8013c840 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.012+0000 7fbd8567d640 10 monclient(hunting): authenticate will time out at 1538.127687s 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 librados: starting msgr at 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 librados: starting objecter 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 Processor -- start 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 -- start start 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 librados: setting wanted keys 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 1 librados: calling monclient init 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.013+0000 7ff5af451640 10 monclient: init 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5a80b0120 0x7ff5a8141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a80b0120 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0d20 0x7ff5a80ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a80b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 0x7ff5a8139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a8144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80aca00 con 0x7ff5a80b0120 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.014+0000 7ff5af451640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80ac430 con 0x7ff5a80b0d20 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.015+0000 7ff5af451640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.015+0000 7ff5af451640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80ac5f0 con 0x7ff5a8144ab0 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.015+0000 7ff5af451640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.015+0000 7ff5af451640 10 monclient(hunting): authenticate will time out at 1538.130196s 2024-03-30T07:16:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 0x7ff5a8139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 0x7ff5a8139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54826/0 (socket says 172.21.15.47:54826) 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 1 -- 172.21.15.47:0/3943223548 learned_addr learned my addr 172.21.15.47:0/3943223548 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 10 monclient(hunting): get_auth_request con 0x7ff5a8144ab0 auth_method 0 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.016+0000 7ff5ad9ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.017+0000 7ff5ad9ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.017+0000 7ff5ad9ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.017+0000 7ff5ad9ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.017+0000 7fbd77fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 0x7fbd8006acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 -- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0d20 msgr2=0x7ff5a80ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7ff5a80b0d20 0x7ff5a80ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 -- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5a80b0120 msgr2=0x7ff5a8141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff5a80b0120 0x7ff5a8141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7ff5ad9ee640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4009400 con 0x7ff5a8144ab0 2024-03-30T07:16:52.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 0x7fbd8006acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54810/0 (socket says 172.21.15.47:54810) 2024-03-30T07:16:52.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 1 -- 172.21.15.47:0/2696313573 learned_addr learned my addr 172.21.15.47:0/2696313573 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 10 monclient(hunting): get_auth_request con 0x7fbd8013c840 auth_method 0 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.020+0000 7fbd77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.023+0000 7fbd77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.023+0000 7fbd77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.023+0000 7fbd77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89140b4a50 con 0x7f890c15c9b0 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140b4dc0 con 0x7f890c15c9b0 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 10 monclient: dump: 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.093 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89140ad400 con 0x7f890c15c9b0 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.025+0000 7f890affd640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f88f806c5c0 0x7f88f806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.026+0000 7f89189b3640 4 set_mon_vals no callback set 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.026+0000 7f891a1b6640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f88f806c5c0 0x7f88f806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.026+0000 7f891a1b6640 10 monclient: get_auth_request con 0x7f88f806c5c0 auth_method 0 2024-03-30T07:16:52.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.026+0000 7f890affd640 10 client.4293.objecter ms_handle_connect 0x7f88f806c5c0 2024-03-30T07:16:52.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.026+0000 7f891a1b6640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f88f806c5c0 0x7f88f806ea80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8910001560 tx=0x7f8910007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-03-30T07:16:52.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 1 -- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd8013fb40 msgr2=0x7fbd801d9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbd8013fb40 0x7fbd801d9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 1 -- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd8006fa60 msgr2=0x7fbd80139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbd8006fa60 0x7fbd80139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.027+0000 7fbd77fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.028+0000 7fbd77fff640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.028+0000 7fbd77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.028+0000 7fbd77fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.028+0000 7fbd77fff640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700094d0 con 0x7fbd8013c840 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.031+0000 7ff5ad9ee640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 0x7ff5a8139fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c780 tx=0x7ff5a000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.031+0000 7ff59dffb640 10 client.?.objecter ms_handle_connect 0x7ff5a8144ab0 2024-03-30T07:16:52.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.031+0000 7ff59dffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.032+0000 7fbd777fe640 10 client.?.objecter ms_handle_connect 0x7fbd8013c840 2024-03-30T07:16:52.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.032+0000 7fbd777fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.032+0000 7fbd77fff640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 0x7fbd8006acf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800c3f0 tx=0x7fbd6800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.032+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6be8003c60 con 0x7f6bf8065f80 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.033+0000 7f6bf57fa640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6be406c5e0 0x7f6be406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.033+0000 7f6bf7fff640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6be406c5e0 0x7f6be406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.033+0000 7f6bf7fff640 10 monclient: get_auth_request con 0x7f6be406c5e0 auth_method 0 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.033+0000 7f6bf7fff640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6be406c5e0 0x7f6be406eaa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6bf813e3c0 tx=0x7f6bec008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.033+0000 7f6bf57fa640 10 client.4315.objecter ms_handle_connect 0x7f6be406c5e0 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6be8057f70 con 0x7f6bf8065f80 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 10 client.4315.objecter ms_dispatch 0x7f6bf8000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 3 client.4315.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 3 client.4315.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 10 client.4315.objecter checking command 1 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7f6be40729c0 0x7f6be4074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter _get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 15 client.4315.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter _get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 15 client.4315.objecter _session_command_op_assign 5 1 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 4 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 10 client.4315.objecter _send_command 1 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be4075490 con 0x7f6be40729c0 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.035+0000 7f6bf57fa640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524040670 con 0x7f65340707d0 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: dump: 2024-03-30T07:16:52.101 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.101 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65240407d0 con 0x7f65340707d0 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6524040b80 con 0x7f65340707d0 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f65317fa640 10 monclient: dump: 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.103 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 5 monclient: authenticate success, global_id 4318 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 10 monclient: _renew_subs 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65340a84a0 con 0x7f65340707d0 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6532ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 10 monclient: _renew_subs 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65341da6a0 con 0x7f65340707d0 2024-03-30T07:16:52.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6539c34640 1 librados: init done 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6bfd746640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7f6be40729c0 0x7f6be4074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.036+0000 7f6bfd746640 10 monclient: get_auth_request con 0x7f6be40729c0 auth_method 0 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 15 client.4318.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.037+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f65340b0130 con 0x7f65340707d0 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0004b60 con 0x7ff5a8144ab0 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: dump: 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.107 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a000ce90 con 0x7ff5a8144ab0 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.038+0000 7ff59f7fe640 4 set_mon_vals no callback set 2024-03-30T07:16:52.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a000a850 con 0x7ff5a8144ab0 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff5af451640 5 monclient: authenticate success, global_id 4244 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff5af451640 10 monclient: _renew_subs 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff5af451640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff5af451640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a813a510 con 0x7ff5a8144ab0 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff59dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff59dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.039+0000 7ff59dffb640 10 monclient: dump: 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.109 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7ff5af451640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7ff5af451640 10 monclient: _renew_subs 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7ff5af451640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7ff5af451640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a813a9d0 con 0x7ff5a8144ab0 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7ff5af451640 1 librados: init done 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 10 client.4244.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 15 client.4244.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 20 client.4244.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5a80b1170 con 0x7ff5a8144ab0 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd680035f0 con 0x7fbd8013c840 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: dump: 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.111 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd68003770 con 0x7fbd8013c840 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68003b20 con 0x7fbd8013c840 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd777fe640 10 monclient: dump: 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.112 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 5 monclient: authenticate success, global_id 4247 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 10 monclient: _renew_subs 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd80140150 con 0x7fbd8013c840 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd7dffb640 4 set_mon_vals no callback set 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 10 monclient: _renew_subs 2024-03-30T07:16:52.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd80069ce0 con 0x7fbd8013c840 2024-03-30T07:16:52.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.040+0000 7fbd8567d640 1 librados: init done 2024-03-30T07:16:52.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff518010ca0 con 0x7ff52406d9b0 2024-03-30T07:16:52.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7f6bfd746640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7f6be40729c0 0x7f6be4074de0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7f6bf57fa640 10 client.4315.objecter ms_handle_connect 0x7f6be40729c0 2024-03-30T07:16:52.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 10 client.4247.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff5217fa640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff50806c5e0 0x7ff50806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 15 client.4247.objecter _session_command_op_assign -1 1 2024-03-30T07:16:52.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 20 client.4247.objecter _get_session osd=-1 returning homeless 2024-03-30T07:16:52.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd8013ffd0 con 0x7fbd8013c840 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff523fff640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff50806c5e0 0x7ff50806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.041+0000 7ff523fff640 10 monclient: get_auth_request con 0x7ff50806c5e0 auth_method 0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.042+0000 7ff5217fa640 10 client.4290.objecter ms_handle_connect 0x7ff50806c5e0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.042+0000 7ff523fff640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff50806c5e0 0x7ff50806eaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5240b2a90 tx=0x7ff510007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.045+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff518058a60 con 0x7ff52406d9b0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.045+0000 7ff5217fa640 10 client.4290.objecter ms_dispatch 0x7ff524000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.045+0000 7ff5217fa640 3 client.4290.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.045+0000 7ff5217fa640 3 client.4290.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 10 client.4290.objecter checking command 1 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7ff5080729c0 0x7ff508074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter _get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 15 client.4290.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter _get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 15 client.4290.objecter _session_command_op_assign 4 1 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 4 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 10 client.4290.objecter _send_command 1 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff508075490 con 0x7ff5080729c0 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5180546a0 con 0x7ff52406d9b0 2024-03-30T07:16:52.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.046+0000 7ff5217fa640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f89140c32a0 con 0x7f890c15c9b0 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 10 client.4293.objecter ms_dispatch 0x7f8914000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 3 client.4293.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 3 client.4293.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 10 client.4293.objecter checking command 1 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f88f80729a0 0x7f88f8074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter _get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 15 client.4293.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter _get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 15 client.4293.objecter _session_command_op_assign 7 1 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 4 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 10 client.4293.objecter _send_command 1 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88f8079880 con 0x7f88f80729a0 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89140f1870 con 0x7f890c15c9b0 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.050+0000 7f890affd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.051+0000 7f89199b5640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f88f80729a0 0x7f88f8074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.052+0000 7f89199b5640 10 monclient: get_auth_request con 0x7f88f80729a0 auth_method 0 2024-03-30T07:16:52.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.057+0000 7f890affd640 10 client.4293.objecter ms_handle_connect 0x7f88f80729a0 2024-03-30T07:16:52.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.057+0000 7f89199b5640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f88f80729a0 0x7f88f8074dc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.059+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f652403c070 con 0x7f65340707d0 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.060+0000 7f65317fa640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f652006c590 0x7f652006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.060+0000 7f6538c32640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f652006c590 0x7f652006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.060+0000 7f6538c32640 10 monclient: get_auth_request con 0x7f652006c590 auth_method 0 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.060+0000 7f65317fa640 10 client.4318.objecter ms_handle_connect 0x7f652006c590 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.060+0000 7f6538c32640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f652006c590 0x7f652006ea50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6528000f60 tx=0x7f65280079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7ff52931b640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7ff5080729c0 0x7ff508074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6524033340 con 0x7f65340707d0 2024-03-30T07:16:52.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 10 client.4318.objecter ms_dispatch 0x7f6534000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 3 client.4318.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 3 client.4318.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 10 client.4318.objecter checking command 1 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f6520072970 0x7f6520074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter _get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 15 client.4318.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter _get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 15 client.4318.objecter _session_command_op_assign 6 1 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 4 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 10 client.4318.objecter _send_command 1 2024-03-30T07:16:52.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6520075440 con 0x7f6520072970 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.063+0000 7f65317fa640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.064+0000 7f6539433640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f6520072970 0x7f6520074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.065+0000 7f6539433640 10 monclient: get_auth_request con 0x7f6520072970 auth_method 0 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.067+0000 7ff52931b640 10 monclient: get_auth_request con 0x7ff5080729c0 auth_method 0 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== osd.3 v2:172.21.15.47:6810/35206 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6c7c075440 con 0x7f6c7c072970 2024-03-30T07:16:52.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6c8d7fa640 10 client.4300.objecter ms_dispatch 0x7f6c90000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6c8d7fa640 10 client.4300.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6c8d7fa640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 4 2024-03-30T07:16:52.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6c8d7fa640 15 client.4300.objecter _session_command_op_remove 3 1 2024-03-30T07:16:52.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f6539433640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f6520072970 0x7f6520074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.070+0000 7f65317fa640 10 client.4318.objecter ms_handle_connect 0x7f6520072970 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.072+0000 7ff52931b640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7ff5080729c0 0x7ff508074de0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.072+0000 7ff5217fa640 10 client.4290.objecter ms_handle_connect 0x7ff5080729c0 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.086+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== osd.0 v2:172.21.15.47:6800/35205 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6a6c0754a0 con 0x7f6a6c0729d0 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.086+0000 7f6a67fff640 10 client.4232.objecter ms_dispatch 0x7f6a80000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.086+0000 7f6a67fff640 10 client.4232.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.086+0000 7f6a67fff640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 4 2024-03-30T07:16:52.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.086+0000 7f6a67fff640 15 client.4232.objecter _session_command_op_remove 0 1 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.087+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5a000a9b0 con 0x7ff5a8144ab0 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.087+0000 7ff59dffb640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff58c06c610 0x7ff58c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.087+0000 7ff5ad1ed640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff58c06c610 0x7ff58c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.087+0000 7ff5ad1ed640 10 monclient: get_auth_request con 0x7ff58c06c610 auth_method 0 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.088+0000 7ff5ad1ed640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff58c06c610 0x7ff58c06ead0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4036000 tx=0x7ff5a4030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.088+0000 7ff59dffb640 10 client.4244.objecter ms_handle_connect 0x7ff58c06c610 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 15 client.4300.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 20 client.4300.objecter _get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 20 client.4300.objecter _get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 15 client.4300.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 20 client.4300.objecter get_session s=0x7f6c7c0722a0 osd=3 3 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 15 client.4300.objecter _session_command_op_assign 3 2 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 4 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 10 client.4300.objecter _send_command 2 2024-03-30T07:16:52.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.099+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 --> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6c900a92d0 con 0x7f6c7c072970 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 10 client.4232.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 15 client.4232.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 20 client.4232.objecter _get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 20 client.4232.objecter _get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 15 client.4232.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 20 client.4232.objecter get_session s=0x7f6a6c072300 osd=0 3 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 15 client.4232.objecter _session_command_op_assign 0 2 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 4 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 10 client.4232.objecter _send_command 2 2024-03-30T07:16:52.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.101+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 --> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a80140160 con 0x7f6a6c0729d0 2024-03-30T07:16:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.109+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbd68003c80 con 0x7fbd8013c840 2024-03-30T07:16:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.110+0000 7fbd777fe640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fbd6006c5e0 0x7fbd6006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.110+0000 7fbd7effd640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fbd6006c5e0 0x7fbd6006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.110+0000 7fbd7effd640 10 monclient: get_auth_request con 0x7fbd6006c5e0 auth_method 0 2024-03-30T07:16:52.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.111+0000 7fbd777fe640 10 client.4247.objecter ms_handle_connect 0x7fbd6006c5e0 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.111+0000 7fbd7effd640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fbd6006c5e0 0x7fbd6006eaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd70036000 tx=0x7fbd70030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c8d7fa640 1 -- 172.21.15.47:0/2524618362 <== osd.3 v2:172.21.15.47:6810/35206 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6c900a92d0 con 0x7f6c7c072970 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c8d7fa640 10 client.4300.objecter ms_dispatch 0x7f6c90000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c8d7fa640 10 client.4300.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c8d7fa640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 4 2024-03-30T07:16:52.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c8d7fa640 15 client.4300.objecter _session_command_op_remove 3 2 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbd68058660 con 0x7fbd8013c840 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 10 client.4247.objecter ms_dispatch 0x7fbd80000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 3 client.4247.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 3 client.4247.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 10 client.4247.objecter checking command 1 2024-03-30T07:16:52.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fbd600729c0 0x7fbd60074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter _get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 15 client.4247.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter _get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 15 client.4247.objecter _session_command_op_assign 2 1 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 4 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 10 client.4247.objecter _send_command 1 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd60075490 con 0x7fbd600729c0 2024-03-30T07:16:52.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7fbd777fe640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c96dc4640 10 librados: watch_flush enter 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c96dc4640 10 librados: watch_flush exit 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c96dc4640 10 client.4300.objecter close_session for osd.3 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f6c7c072970 msgr2=0x7f6c7c074d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.114+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f6c7c072970 0x7f6c7c074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 20 client.4300.objecter put_session s=0x7f6c7c0722a0 osd=3 1 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 10 client.4300.objecter successfully canceled tick 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6c7c06c590 msgr2=0x7f6c7c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6c7c06c590 0x7f6c7c06ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6c8003a000 tx=0x7f6c80032000 comp rx=0 tx=0).stop 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 10 monclient: shutdown 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 20 monclient: shutdown canceling and discarding version request 1 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006d830 msgr2=0x7f6c900a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 --2- 172.21.15.47:0/2524618362 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6c9006d830 0x7f6c900a9f00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c740 tx=0x7f6c8800cc00 comp rx=0 tx=0).stop 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c95361640 1 -- 172.21.15.47:0/2524618362 reap_dead start 2024-03-30T07:16:52.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 shutdown_connections 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 >> 172.21.15.47:0/2524618362 conn(0x7f6c900b2cb0 msgr2=0x7f6c900aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 shutdown_connections 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 -- 172.21.15.47:0/2524618362 wait complete. 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7f6c96dc4640 1 librados: shutdown 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.115+0000 7fbd7f7fe640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fbd600729c0 0x7fbd60074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.116+0000 7fbd7f7fe640 10 monclient: get_auth_request con 0x7fbd600729c0 auth_method 0 2024-03-30T07:16:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a67fff640 1 -- 172.21.15.47:0/4198320581 <== osd.0 v2:172.21.15.47:6800/35205 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6a7802cd80 con 0x7f6a6c0729d0 2024-03-30T07:16:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a67fff640 10 client.4232.objecter ms_dispatch 0x7f6a80000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a67fff640 10 client.4232.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a67fff640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 4 2024-03-30T07:16:52.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a67fff640 15 client.4232.objecter _session_command_op_remove 0 2 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 10 librados: watch_flush enter 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 10 librados: watch_flush exit 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 10 client.4232.objecter close_session for osd.0 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6a6c0729d0 msgr2=0x7f6a6c074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6a6c0729d0 0x7f6a6c074df0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 20 client.4232.objecter put_session s=0x7f6a6c072300 osd=0 1 2024-03-30T07:16:52.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 10 client.4232.objecter successfully canceled tick 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6a6c06c5f0 msgr2=0x7f6a6c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6a6c06c5f0 0x7f6a6c06eab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000f60 tx=0x7f6a70009040 comp rx=0 tx=0).stop 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 10 monclient: shutdown 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 msgr2=0x7f6a8013e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 --2- 172.21.15.47:0/4198320581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6a801491c0 0x7f6a8013e7e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400b440 tx=0x7f6a7400b900 comp rx=0 tx=0).stop 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a7ffff640 1 -- 172.21.15.47:0/4198320581 reap_dead start 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 shutdown_connections 2024-03-30T07:16:52.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.119+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 >> 172.21.15.47:0/4198320581 conn(0x7f6a800b2cb0 msgr2=0x7f6a800701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.120+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 shutdown_connections 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.120+0000 7f6a85c41640 1 -- 172.21.15.47:0/4198320581 wait complete. 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.120+0000 7f6a85c41640 1 librados: shutdown 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== osd.5 v2:172.21.15.52:6826/35314 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6be4075490 con 0x7f6be40729c0 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f6bf57fa640 10 client.4315.objecter ms_dispatch 0x7f6bf8000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f6bf57fa640 10 client.4315.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f6bf57fa640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 4 2024-03-30T07:16:52.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f6bf57fa640 15 client.4315.objecter _session_command_op_remove 5 1 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== osd.6 v2:172.21.15.52:6802/35315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6520075440 con 0x7f6520072970 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f65317fa640 10 client.4318.objecter ms_dispatch 0x7f6534000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f65317fa640 10 client.4318.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f65317fa640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 4 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.121+0000 7f65317fa640 15 client.4318.objecter _session_command_op_remove 6 1 2024-03-30T07:16:52.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff5a00591c0 con 0x7ff5a8144ab0 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 10 client.4244.objecter ms_dispatch 0x7ff5a8000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 3 client.4244.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 3 client.4244.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 10 client.4244.objecter checking command 1 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7ff58c0729f0 0x7ff58c074e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter _get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 15 client.4244.objecter _session_command_op_remove -1 1 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter _get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 15 client.4244.objecter _session_command_op_assign 1 1 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 4 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 10 client.4244.objecter _send_command 1 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff58c0754c0 con 0x7ff58c0729f0 2024-03-30T07:16:52.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.123+0000 7ff59dffb640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.124+0000 7ff5ac9ec640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7ff58c0729f0 0x7ff58c074e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.125+0000 7ff5ac9ec640 10 monclient: get_auth_request con 0x7ff58c0729f0 auth_method 0 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.126+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== osd.7 v2:172.21.15.52:6804/35316 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f88f8079880 con 0x7f88f80729a0 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.126+0000 7f890affd640 10 client.4293.objecter ms_dispatch 0x7f8914000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.126+0000 7f890affd640 10 client.4293.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.126+0000 7f890affd640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 4 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.126+0000 7f890affd640 15 client.4293.objecter _session_command_op_remove 7 1 2024-03-30T07:16:52.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.129+0000 7ff5ac9ec640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7ff58c0729f0 0x7ff58c074e10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.129+0000 7ff59dffb640 10 client.4244.objecter ms_handle_connect 0x7ff58c0729f0 2024-03-30T07:16:52.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.130+0000 7fbd777fe640 10 client.4247.objecter ms_handle_connect 0x7fbd600729c0 2024-03-30T07:16:52.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.130+0000 7fbd7f7fe640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fbd600729c0 0x7fbd60074de0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.148 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-03-30T07:16:52.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T07:16:52.149 INFO:teuthology.orchestra.run.smithi047.stdout:94489280522 2024-03-30T07:16:52.149 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 15 client.4293.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 20 client.4293.objecter _get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 20 client.4293.objecter _get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 15 client.4293.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 20 client.4293.objecter get_session s=0x7f88f80722d0 osd=7 3 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 15 client.4293.objecter _session_command_op_assign 7 2 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 4 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 10 client.4293.objecter _send_command 2 2024-03-30T07:16:52.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.145+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 --> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f890c033880 con 0x7f88f80729a0 2024-03-30T07:16:52.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 10 client.4315.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 15 client.4315.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 20 client.4315.objecter _get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 20 client.4315.objecter _get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 15 client.4315.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 20 client.4315.objecter get_session s=0x7f6be40722f0 osd=5 3 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 15 client.4315.objecter _session_command_op_assign 5 2 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 4 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 10 client.4315.objecter _send_command 2 2024-03-30T07:16:52.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.147+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 --> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6bf804e030 con 0x7f6be40729c0 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 15 client.4318.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 20 client.4318.objecter _get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 20 client.4318.objecter _get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 15 client.4318.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 20 client.4318.objecter get_session s=0x7f65200722a0 osd=6 3 2024-03-30T07:16:52.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 15 client.4318.objecter _session_command_op_assign 6 2 2024-03-30T07:16:52.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 4 2024-03-30T07:16:52.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 10 client.4318.objecter _send_command 2 2024-03-30T07:16:52.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.149+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f653404e030 con 0x7f6520072970 2024-03-30T07:16:52.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f890affd640 1 -- 172.21.15.47:0/728663371 <== osd.7 v2:172.21.15.52:6804/35316 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8904005a80 con 0x7f88f80729a0 2024-03-30T07:16:52.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f890affd640 10 client.4293.objecter ms_dispatch 0x7f8914000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f890affd640 10 client.4293.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f890affd640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 4 2024-03-30T07:16:52.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f890affd640 15 client.4293.objecter _session_command_op_remove 7 2 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 10 librados: watch_flush enter 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 10 librados: watch_flush exit 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 10 client.4293.objecter close_session for osd.7 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f88f80729a0 msgr2=0x7f88f8074dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f88f80729a0 0x7f88f8074dc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 20 client.4293.objecter put_session s=0x7f88f80722d0 osd=7 1 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 10 client.4293.objecter successfully canceled tick 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f88f806c5c0 msgr2=0x7f88f806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f88f806c5c0 0x7f88f806ea80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8910001560 tx=0x7f8910007990 comp rx=0 tx=0).stop 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 10 monclient: shutdown 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 msgr2=0x7f890c0ba240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 --2- 172.21.15.47:0/728663371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f890c15c9b0 0x7f890c0ba240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8914063b00 tx=0x7f8914068530 comp rx=0 tx=0).stop 2024-03-30T07:16:52.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891a9b7640 1 -- 172.21.15.47:0/728663371 reap_dead start 2024-03-30T07:16:52.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 shutdown_connections 2024-03-30T07:16:52.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.153+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 >> 172.21.15.47:0/728663371 conn(0x7f890c006510 msgr2=0x7f890c004d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.155+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6be80542b0 con 0x7f6bf8065f80 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.155+0000 7f6bf57fa640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f6bf57fa640 1 -- 172.21.15.47:0/3643531402 <== osd.5 v2:172.21.15.52:6826/35314 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6bf804e030 con 0x7f6be40729c0 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f6bf57fa640 10 client.4315.objecter ms_dispatch 0x7f6bf8000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f6bf57fa640 10 client.4315.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f6bf57fa640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 4 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f6bf57fa640 15 client.4315.objecter _session_command_op_remove 5 2 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6524077a60 con 0x7f65340707d0 2024-03-30T07:16:52.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.156+0000 7f65317fa640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.155+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 shutdown_connections 2024-03-30T07:16:52.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.155+0000 7f891c41a640 1 -- 172.21.15.47:0/728663371 wait complete. 2024-03-30T07:16:52.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.155+0000 7f891c41a640 1 librados: shutdown 2024-03-30T07:16:52.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f65317fa640 1 -- 172.21.15.47:0/2917011029 <== osd.6 v2:172.21.15.52:6802/35315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f653404e030 con 0x7f6520072970 2024-03-30T07:16:52.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f65317fa640 10 client.4318.objecter ms_dispatch 0x7f6534000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f65317fa640 10 client.4318.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f65317fa640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 4 2024-03-30T07:16:52.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f65317fa640 15 client.4318.objecter _session_command_op_remove 6 2 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 10 librados: watch_flush enter 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 10 librados: watch_flush exit 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 10 client.4315.objecter close_session for osd.5 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7f6be40729c0 msgr2=0x7f6be4074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7f6be40729c0 0x7f6be4074de0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 20 client.4315.objecter put_session s=0x7f6be40722f0 osd=5 1 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 10 client.4315.objecter successfully canceled tick 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6539c34640 10 librados: watch_flush enter 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6be406c5e0 msgr2=0x7f6be406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6be406c5e0 0x7f6be406eaa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6bf813e3c0 tx=0x7f6bec008030 comp rx=0 tx=0).stop 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 10 monclient: shutdown 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bf8065f80 msgr2=0x7f6bf81374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfdf47640 1 --2- 172.21.15.47:0/3643531402 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6bf8065f80 0x7f6bf81374b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6be800c470 tx=0x7f6be800c870 comp rx=0 tx=0).stop 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.157+0000 7f6bfd746640 1 -- 172.21.15.47:0/3643531402 reap_dead start 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 shutdown_connections 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 >> 172.21.15.47:0/3643531402 conn(0x7f6bf80b7120 msgr2=0x7f6bf8138e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 10 librados: watch_flush exit 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 10 client.4318.objecter close_session for osd.6 2024-03-30T07:16:52.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f6520072970 msgr2=0x7f6520074d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 shutdown_connections 2024-03-30T07:16:52.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6bfdf47640 1 -- 172.21.15.47:0/3643531402 wait complete. 2024-03-30T07:16:52.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f6520072970 0x7f6520074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6bfdf47640 1 librados: shutdown 2024-03-30T07:16:52.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 20 client.4318.objecter put_session s=0x7f65200722a0 osd=6 1 2024-03-30T07:16:52.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 10 client.4318.objecter successfully canceled tick 2024-03-30T07:16:52.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f652006c590 msgr2=0x7f652006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f652006c590 0x7f652006ea50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6528000f60 tx=0x7f65280079b0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 10 monclient: shutdown 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 msgr2=0x7f653413def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.158+0000 7f6539c34640 1 --2- 172.21.15.47:0/2917011029 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f65340707d0 0x7f653413def0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6524009300 tx=0x7f65240046c0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539433640 1 -- 172.21.15.47:0/2917011029 reap_dead start 2024-03-30T07:16:52.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 shutdown_connections 2024-03-30T07:16:52.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 >> 172.21.15.47:0/2917011029 conn(0x7f65340af610 msgr2=0x7f65340aa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 shutdown_connections 2024-03-30T07:16:52.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539c34640 1 -- 172.21.15.47:0/2917011029 wait complete. 2024-03-30T07:16:52.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.160+0000 7f6539c34640 1 librados: shutdown 2024-03-30T07:16:52.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.169+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== osd.4 v2:172.21.15.52:6818/35313 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff508075490 con 0x7ff5080729c0 2024-03-30T07:16:52.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.169+0000 7ff5217fa640 10 client.4290.objecter ms_dispatch 0x7ff524000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.169+0000 7ff5217fa640 10 client.4290.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.169+0000 7ff5217fa640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 4 2024-03-30T07:16:52.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.169+0000 7ff5217fa640 15 client.4290.objecter _session_command_op_remove 4 1 2024-03-30T07:16:52.178 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-03-30T07:16:52.178 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T07:16:52.180 INFO:teuthology.orchestra.run.smithi047.stdout:81604378633 2024-03-30T07:16:52.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T07:16:52.181 INFO:teuthology.orchestra.run.smithi047.stdout:60129542153 2024-03-30T07:16:52.182 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T07:16:52.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.189+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== osd.1 v2:172.21.15.47:6808/35204 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff58c0754c0 con 0x7ff58c0729f0 2024-03-30T07:16:52.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.189+0000 7ff59dffb640 10 client.4244.objecter ms_dispatch 0x7ff5a8000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.189+0000 7ff59dffb640 10 client.4244.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.189+0000 7ff59dffb640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 4 2024-03-30T07:16:52.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.189+0000 7ff59dffb640 15 client.4244.objecter _session_command_op_remove 1 1 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 15 client.4290.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 20 client.4290.objecter _get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 20 client.4290.objecter _get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 15 client.4290.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 20 client.4290.objecter get_session s=0x7ff5080722f0 osd=4 3 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 15 client.4290.objecter _session_command_op_assign 4 2 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 4 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 10 client.4290.objecter _send_command 2 2024-03-30T07:16:52.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.195+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 --> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff524071350 con 0x7ff5080729c0 2024-03-30T07:16:52.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff5217fa640 1 -- 172.21.15.47:0/3219478015 <== osd.4 v2:172.21.15.52:6818/35313 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff51c02c7f0 con 0x7ff5080729c0 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff5217fa640 10 client.4290.objecter ms_dispatch 0x7ff524000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff5217fa640 10 client.4290.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff5217fa640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 4 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff5217fa640 15 client.4290.objecter _session_command_op_remove 4 2 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 10 librados: watch_flush enter 2024-03-30T07:16:52.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 10 librados: watch_flush exit 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 10 client.4290.objecter close_session for osd.4 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7ff5080729c0 msgr2=0x7ff508074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7ff5080729c0 0x7ff508074de0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 20 client.4290.objecter put_session s=0x7ff5080722f0 osd=4 1 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.207+0000 7ff52ad7e640 10 client.4290.objecter successfully canceled tick 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff50806c5e0 msgr2=0x7ff50806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff50806c5e0 0x7ff50806eaa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff5240b2a90 tx=0x7ff510007990 comp rx=0 tx=0).stop 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 10 monclient: shutdown 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 msgr2=0x7ff524139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 --2- 172.21.15.47:0/3219478015 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff52406d9b0 0x7ff524139f20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b4a0 tx=0x7ff51800b960 comp rx=0 tx=0).stop 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52931b640 1 -- 172.21.15.47:0/3219478015 reap_dead start 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 shutdown_connections 2024-03-30T07:16:52.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 >> 172.21.15.47:0/3219478015 conn(0x7ff5240b3f70 msgr2=0x7ff524069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.209+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 shutdown_connections 2024-03-30T07:16:52.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.210+0000 7ff52ad7e640 1 -- 172.21.15.47:0/3219478015 wait complete. 2024-03-30T07:16:52.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.210+0000 7ff52ad7e640 1 librados: shutdown 2024-03-30T07:16:52.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.223+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5a0054e00 con 0x7ff5a8144ab0 2024-03-30T07:16:52.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.223+0000 7ff59dffb640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.223+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd680542a0 con 0x7fbd8013c840 2024-03-30T07:16:52.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.223+0000 7fbd777fe640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:16:52.230 INFO:teuthology.orchestra.run.smithi047.stdout:85899345930 2024-03-30T07:16:52.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T07:16:52.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 10 client.4244.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 15 client.4244.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 20 client.4244.objecter _get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 20 client.4244.objecter _get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 15 client.4244.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 20 client.4244.objecter get_session s=0x7ff58c072320 osd=1 3 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 15 client.4244.objecter _session_command_op_assign 1 2 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 4 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 10 client.4244.objecter _send_command 2 2024-03-30T07:16:52.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.229+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 --> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5a80ae1c0 con 0x7ff58c0729f0 2024-03-30T07:16:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.239+0000 7ff59dffb640 1 -- 172.21.15.47:0/3943223548 <== osd.1 v2:172.21.15.47:6808/35204 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff5a80ae1c0 con 0x7ff58c0729f0 2024-03-30T07:16:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.239+0000 7ff59dffb640 10 client.4244.objecter ms_dispatch 0x7ff5a8000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.239+0000 7ff59dffb640 10 client.4244.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.239+0000 7ff59dffb640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 4 2024-03-30T07:16:52.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.239+0000 7ff59dffb640 15 client.4244.objecter _session_command_op_remove 1 2 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 10 librados: watch_flush enter 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 10 librados: watch_flush exit 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 10 client.4244.objecter close_session for osd.1 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7ff58c0729f0 msgr2=0x7ff58c074e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7ff58c0729f0 0x7ff58c074e10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 20 client.4244.objecter put_session s=0x7ff58c072320 osd=1 1 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 10 client.4244.objecter successfully canceled tick 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff58c06c610 msgr2=0x7ff58c06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7ff58c06c610 0x7ff58c06ead0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4036000 tx=0x7ff5a4030000 comp rx=0 tx=0).stop 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 10 monclient: shutdown 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 msgr2=0x7ff5a8139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff58b7fe640 1 --2- 172.21.15.47:0/3943223548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff5a8144ab0 0x7ff5a8139fd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c780 tx=0x7ff5a000cc40 comp rx=0 tx=0).stop 2024-03-30T07:16:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.240+0000 7ff5ad9ee640 1 -- 172.21.15.47:0/3943223548 reap_dead start 2024-03-30T07:16:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.241+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 shutdown_connections 2024-03-30T07:16:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.241+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 >> 172.21.15.47:0/3943223548 conn(0x7ff5a80b70b0 msgr2=0x7ff5a80b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.241+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 shutdown_connections 2024-03-30T07:16:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.241+0000 7ff58b7fe640 1 -- 172.21.15.47:0/3943223548 wait complete. 2024-03-30T07:16:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.242+0000 7ff58b7fe640 1 librados: shutdown 2024-03-30T07:16:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.247+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== osd.2 v2:172.21.15.47:6824/35207 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbd60075490 con 0x7fbd600729c0 2024-03-30T07:16:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.247+0000 7fbd777fe640 10 client.4247.objecter ms_dispatch 0x7fbd80000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:16:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.247+0000 7fbd777fe640 10 client.4247.objecter _finish_command 1 = system:0 2024-03-30T07:16:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.247+0000 7fbd777fe640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 4 2024-03-30T07:16:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.247+0000 7fbd777fe640 15 client.4247.objecter _session_command_op_remove 2 1 2024-03-30T07:16:52.261 INFO:teuthology.orchestra.run.smithi047.stdout:68719476744 2024-03-30T07:16:52.261 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T07:16:52.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 10 client.4247.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-30T07:16:52.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 15 client.4247.objecter _session_command_op_assign -1 2 2024-03-30T07:16:52.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 20 client.4247.objecter _get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 20 client.4247.objecter _get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 15 client.4247.objecter _session_command_op_remove -1 2 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 20 client.4247.objecter get_session s=0x7fbd600722f0 osd=2 3 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 15 client.4247.objecter _session_command_op_assign 2 2 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 4 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 10 client.4247.objecter _send_command 2 2024-03-30T07:16:52.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.277+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 --> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd800af3f0 con 0x7fbd600729c0 2024-03-30T07:16:52.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd777fe640 1 -- 172.21.15.47:0/2696313573 <== osd.2 v2:172.21.15.47:6824/35207 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbd7800f040 con 0x7fbd600729c0 2024-03-30T07:16:52.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd777fe640 10 client.4247.objecter ms_dispatch 0x7fbd80000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:16:52.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd777fe640 10 client.4247.objecter _finish_command 2 = system:0 2024-03-30T07:16:52.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd777fe640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 4 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd777fe640 15 client.4247.objecter _session_command_op_remove 2 2 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 10 librados: watch_flush enter 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 10 librados: watch_flush exit 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 10 client.4247.objecter close_session for osd.2 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fbd600729c0 msgr2=0x7fbd60074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fbd600729c0 0x7fbd60074de0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 20 client.4247.objecter put_session s=0x7fbd600722f0 osd=2 1 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-30T07:16:52.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.295+0000 7fbd8567d640 10 client.4247.objecter successfully canceled tick 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fbd6006c5e0 msgr2=0x7fbd6006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fbd6006c5e0 0x7fbd6006eaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbd70036000 tx=0x7fbd70030000 comp rx=0 tx=0).stop 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 10 monclient: shutdown 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 msgr2=0x7fbd8006acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 --2- 172.21.15.47:0/2696313573 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbd8013c840 0x7fbd8006acf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800c3f0 tx=0x7fbd6800c8b0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd7f7fe640 1 -- 172.21.15.47:0/2696313573 reap_dead start 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 shutdown_connections 2024-03-30T07:16:52.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.296+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 >> 172.21.15.47:0/2696313573 conn(0x7fbd800b4e40 msgr2=0x7fbd800aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.298+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 shutdown_connections 2024-03-30T07:16:52.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.298+0000 7fbd8567d640 1 -- 172.21.15.47:0/2696313573 wait complete. 2024-03-30T07:16:52.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.299+0000 7fbd8567d640 1 librados: shutdown 2024-03-30T07:16:52.320 INFO:teuthology.orchestra.run.smithi047.stdout:94489280523 2024-03-30T07:16:52.320 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.325+0000 7f31349ab640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.325+0000 7f31349ab640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.325+0000 7f31349ab640 10 monclient: monmap: 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.326473+0000 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.326473+0000 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.327 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.328 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.328 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.326+0000 7f31349ab640 1 Processor -- start 2024-03-30T07:16:52.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.327+0000 7f31349ab640 1 -- start start 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.327+0000 7f31349ab640 10 monclient: init 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.328+0000 7f31349ab640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.328+0000 7f31349ab640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.328+0000 7f31349ab640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f31300b0120 0x7f31300a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.328+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-a con 0x7f31300b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3130144ab0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f31300b0d20 0x7f313014be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-c con 0x7f31300b0d20 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300a8bd0 con 0x7f3130144ab0 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31300aa4f0 con 0x7f31300b0120 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3130144f10 con 0x7f31300b0d20 2024-03-30T07:16:52.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.329+0000 7f31349ab640 10 monclient(hunting): authenticate will time out at 1538.444200s 2024-03-30T07:16:52.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f31300b0d20 0x7f313014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f31300b0d20 0x7f313014be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44130/0 (socket says 172.21.15.47:44130) 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 1 -- 172.21.15.47:0/1307700069 learned_addr learned my addr 172.21.15.47:0/1307700069 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 10 monclient(hunting): get_auth_request con 0x7f31300b0d20 auth_method 0 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.331+0000 7f312e59c640 1 --2- 172.21.15.47:0/1307700069 >> v2:172.21.15.47:3300/0 conn(0x7f31300b0120 0x7f31300a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.332+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.332+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.332+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.333+0000 7f312e59c640 10 monclient(hunting): get_auth_request con 0x7f31300b0120 auth_method 0 2024-03-30T07:16:52.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.333+0000 7f312e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.333+0000 7f312e59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.333+0000 7f312e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 1 -- 172.21.15.47:0/1307700069 >> v2:172.21.15.47:3300/0 conn(0x7f31300b0120 msgr2=0x7f31300a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 1 --2- 172.21.15.47:0/1307700069 >> v2:172.21.15.47:3300/0 conn(0x7f31300b0120 0x7f31300a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 1 -- 172.21.15.47:0/1307700069 >> v1:172.21.15.47:6789/0 conn(0x7f3130144ab0 legacy=0x7f31300712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.337+0000 7f312ed9d640 1 -- 172.21.15.47:0/1307700069 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300b3780 con 0x7f31300b0d20 2024-03-30T07:16:52.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.341+0000 7f312ed9d640 1 --2- 172.21.15.47:0/1307700069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f313014be20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31300b2b00 tx=0x7f312802bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c2c0610cbd4e2e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.342+0000 7f13bac70640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.342+0000 7f13bac70640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.342+0000 7f13bac70640 10 monclient: monmap: 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.343507+0000 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.343507+0000 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.343 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.343+0000 7f13bac70640 1 Processor -- start 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.343+0000 7f13bac70640 1 -- start start 2024-03-30T07:16:52.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.343+0000 7f13bac70640 10 monclient: init 2024-03-30T07:16:52.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 1 -- 172.21.15.47:0/1307700069 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312802cb30 con 0x7f31300b0d20 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: dump: 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.346 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 1 -- 172.21.15.47:0/1307700069 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312802cc90 con 0x7f31300b0d20 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 1 -- 172.21.15.47:0/1307700069 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128034610 con 0x7f31300b0d20 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f312dd9b640 10 monclient: dump: 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.348 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.349 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-03-30T07:16:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f13b40b0f50 0x7f13b4142d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-c con 0x7f13b40b0f50 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:52.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f13b40b27c0 0x7f13b4143250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.344+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-a con 0x7f13b40b27c0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13b4149330 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b41459b0 con 0x7f13b4149330 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b40b1330 con 0x7f13b40b27c0 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b4149eb0 con 0x7f13b40b0f50 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.345+0000 7f13bac70640 10 monclient(hunting): authenticate will time out at 1538.460159s 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f31349ab640 5 monclient: authenticate success, global_id 4321 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f31349ab640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f31349ab640 4 set_mon_vals no callback set 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 10 monclient: shutdown 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 -- 172.21.15.47:0/1307700069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 msgr2=0x7f313014be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 --2- 172.21.15.47:0/1307700069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f313014be20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31300b2b00 tx=0x7f312802bf90 comp rx=0 tx=0).stop 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 -- 172.21.15.47:0/1307700069 shutdown_connections 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 --2- 172.21.15.47:0/1307700069 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f313014be20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 --2- 172.21.15.47:0/1307700069 >> v2:172.21.15.47:3300/0 conn(0x7f31300b0120 0x7f31300a84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.347+0000 7f31349ab640 1 -- 172.21.15.47:0/1307700069 >> 172.21.15.47:0/1307700069 conn(0x7f31300b70b0 msgr2=0x7f31300b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b920d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f13b4149330 0x7f13b41497a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56256/0 (socket says 172.21.15.47:56256) 2024-03-30T07:16:52.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b920d640 1 -- 172.21.15.47:0/188735217 learned_addr learned my addr 172.21.15.47:0/188735217 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b8a0c640 1 --2- 172.21.15.47:0/188735217 >> v2:172.21.15.52:3300/0 conn(0x7f13b40b0f50 0x7f13b4142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b8a0c640 10 monclient(hunting): get_auth_request con 0x7f13b40b0f50 auth_method 0 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b8a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b8a0c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b8a0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b3fff640 1 --2- 172.21.15.47:0/188735217 >> v2:172.21.15.47:3300/0 conn(0x7f13b40b27c0 0x7f13b4143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b3fff640 10 monclient(hunting): get_auth_request con 0x7f13b40b27c0 auth_method 0 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.346+0000 7f13b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b8a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b8a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b8a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f31349ab640 1 -- 172.21.15.47:0/1307700069 shutdown_connections 2024-03-30T07:16:52.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f13b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-03-30T07:16:52.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 1 -- 172.21.15.47:0/188735217 >> v2:172.21.15.47:3300/0 conn(0x7f13b40b27c0 msgr2=0x7f13b4143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 1 --2- 172.21.15.47:0/188735217 >> v2:172.21.15.47:3300/0 conn(0x7f13b40b27c0 0x7f13b4143250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 1 -- 172.21.15.47:0/188735217 >> v1:172.21.15.52:6789/0 conn(0x7f13b4149330 legacy=0x7f13b41497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.350+0000 7f13b8a0c640 1 -- 172.21.15.47:0/188735217 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4146930 con 0x7f13b40b0f50 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.348+0000 7f31349ab640 1 -- 172.21.15.47:0/1307700069 wait complete. 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.352+0000 7f31349ab640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.352+0000 7f31349ab640 10 monclient: monmap: 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.361 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 librados: starting msgr at 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 librados: starting objecter 2024-03-30T07:16:52.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 Processor -- start 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 -- start start 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 librados: setting wanted keys 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 1 librados: calling monclient init 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.353+0000 7f31349ab640 10 monclient: init 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31300b0120 0x7f31300aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-a con 0x7f31300b0120 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f31300ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-b con 0x7f31300b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): picked mon.noname-c con 0x7f3130144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31300acbd0 con 0x7f31300b0120 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313013e890 con 0x7f31300b0d20 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313013ca50 con 0x7f3130144ab0 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.354+0000 7f31349ab640 10 monclient(hunting): authenticate will time out at 1538.469419s 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.355+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54832/0 (socket says 172.21.15.47:54832) 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312e59c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31300b0120 0x7f31300aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 1 -- 172.21.15.47:0/414862862 learned_addr learned my addr 172.21.15.47:0/414862862 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 10 monclient(hunting): get_auth_request con 0x7f3130144ab0 auth_method 0 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312e59c640 10 monclient(hunting): get_auth_request con 0x7f31300b0120 auth_method 0 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312e59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.356+0000 7f312e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f3127fff640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f31300ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f3127fff640 10 monclient(hunting): get_auth_request con 0x7f31300b0d20 auth_method 0 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f3127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f3127fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f3127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.357+0000 7f312e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.359+0000 7f13b8a0c640 1 --2- 172.21.15.47:0/188735217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b40b0f50 0x7f13b4142d10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13b40b2510 tx=0x7f13a802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d63c2ae6375bc65f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.359+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.359+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.359+0000 7f3127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-03-30T07:16:52.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 -- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 msgr2=0x7f31300ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f31300ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 -- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31300b0120 msgr2=0x7f31300aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31300b0120 0x7f31300aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.361+0000 7f312ed9d640 1 -- 172.21.15.47:0/414862862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3128009b30 con 0x7f3130144ab0 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.365+0000 7f312ed9d640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3128009f40 tx=0x7f3128004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.365+0000 7f3126ffd640 10 client.?.objecter ms_handle_connect 0x7f3130144ab0 2024-03-30T07:16:52.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.365+0000 7f3126ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.367+0000 7f82b59b1640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.367+0000 7f82b59b1640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.367+0000 7f13b37fe640 1 -- 172.21.15.47:0/188735217 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8031870 con 0x7f13b40b0f50 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.367+0000 7f13b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.367+0000 7f82b59b1640 10 monclient: monmap: 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.371 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.368392+0000 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.368392+0000 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.372 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13b37fe640 10 monclient: dump: 2024-03-30T07:16:52.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13b37fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13b37fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13bac70640 5 monclient: authenticate success, global_id 4324 2024-03-30T07:16:52.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f13bac70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7fc8b9570640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7fc8b9570640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7fc8b9570640 10 monclient: monmap: 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.369071+0000 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.369071+0000 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.375 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312803e450 con 0x7f3130144ab0 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 10 monclient: dump: 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f3126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 5 monclient: authenticate success, global_id 4250 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 10 monclient: _renew_subs 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31300af860 con 0x7f3130144ab0 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 10 monclient: _renew_subs 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313013d2f0 con 0x7f3130144ab0 2024-03-30T07:16:52.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f31349ab640 1 librados: init done 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f13b37fe640 1 -- 172.21.15.47:0/188735217 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a80319d0 con 0x7f13b40b0f50 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f13b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f13bac70640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f13bac70640 4 set_mon_vals no callback set 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 10 monclient: shutdown 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 -- 172.21.15.47:0/188735217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b40b0f50 msgr2=0x7f13b4142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 --2- 172.21.15.47:0/188735217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b40b0f50 0x7f13b4142d10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13b40b2510 tx=0x7f13a802b910 comp rx=0 tx=0).stop 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 -- 172.21.15.47:0/188735217 shutdown_connections 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 --2- 172.21.15.47:0/188735217 >> v2:172.21.15.47:3300/0 conn(0x7f13b40b27c0 0x7f13b4143250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 --2- 172.21.15.47:0/188735217 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b40b0f50 0x7f13b4142d10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7f13bac70640 1 -- 172.21.15.47:0/188735217 >> 172.21.15.47:0/188735217 conn(0x7f13b40b70b0 msgr2=0x7f13b40b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f13bac70640 1 -- 172.21.15.47:0/188735217 shutdown_connections 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7fc8b9570640 1 Processor -- start 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7fc8b9570640 1 -- start start 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7fc8b9570640 10 monclient: init 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b406d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b406d290 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8b40aed80 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc8b4145bb0 0x7fc8b4146060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b4145bb0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40ad510 con 0x7fc8b40aed80 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.370+0000 7fc8b9570640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b406e7f0 con 0x7fc8b4145bb0 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7fc8b9570640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7fc8b9570640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b406eaf0 con 0x7fc8b406d290 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7fc8b9570640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7fc8b9570640 10 monclient(hunting): authenticate will time out at 1538.486219s 2024-03-30T07:16:52.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f82b59b1640 1 Processor -- start 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f82b59b1640 1 -- start start 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.368+0000 7f82b59b1640 10 monclient: init 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82b00b0120 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 0x7f82b0144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-b con 0x7f82b00a84a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f82b00b0d20 0x7f82b014be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-a con 0x7f82b00b0d20 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00a8900 con 0x7f82b00b0120 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b00aa4f0 con 0x7f82b00b0d20 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00a8bd0 con 0x7f82b00a84a0 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.369+0000 7f82b59b1640 10 monclient(hunting): authenticate will time out at 1538.485125s 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82aeffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f82b00b0120 0x7f82b00712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56270/0 (socket says 172.21.15.47:56270) 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82aeffd640 1 -- 172.21.15.47:0/1617636130 learned_addr learned my addr 172.21.15.47:0/1617636130 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1617636130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 0x7f82b0144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82ae7fc640 10 monclient(hunting): get_auth_request con 0x7f82b00a84a0 auth_method 0 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82af7fe640 1 --2- 172.21.15.47:0/1617636130 >> v2:172.21.15.47:3300/0 conn(0x7f82b00b0d20 0x7f82b014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82af7fe640 10 monclient(hunting): get_auth_request con 0x7f82b00b0d20 auth_method 0 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.371+0000 7f82af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.372+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.372+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.372+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.372+0000 7f13bac70640 1 -- 172.21.15.47:0/188735217 wait complete. 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 10 monclient: monmap: 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.386 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.387 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f31349ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f31349ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f31349ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313013ea00 con 0x7f3130144ab0 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 librados: starting msgr at 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 librados: starting objecter 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 Processor -- start 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 -- start start 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 librados: setting wanted keys 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 1 librados: calling monclient init 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7f13bac70640 10 monclient: init 2024-03-30T07:16:52.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7fc8b37fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc8b40aed80 0x7fc8b40713d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40790/0 (socket says 172.21.15.47:40790) 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.373+0000 7fc8b37fe640 1 -- 172.21.15.47:0/1449243680 learned_addr learned my addr 172.21.15.47:0/1449243680 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7fc8b8d6f640 1 --2- 172.21.15.47:0/1449243680 >> v2:172.21.15.47:3300/0 conn(0x7fc8b4145bb0 0x7fc8b4146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7fc8b8d6f640 10 monclient(hunting): get_auth_request con 0x7fc8b4145bb0 auth_method 0 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7fc8b8d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7fc8b8d6f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7fc8b8d6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 1 -- 172.21.15.47:0/1617636130 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744703996 0 0) 0x7f82a0001110 con 0x7f82b00b0120 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 10 monclient(hunting): _negotiate 2024-03-30T07:16:52.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 10 monclient(hunting): my global_id is 4333 2024-03-30T07:16:52.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f82adffb640 1 -- 172.21.15.47:0/1617636130 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298003520 con 0x7f82b00b0120 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b40b0f50 0x7f13b4142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-a con 0x7f13b40b0f50 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f13b40b27c0 0x7f13b414c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-c con 0x7f13b40b27c0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.374+0000 7f13bac70640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): picked mon.noname-b con 0x7f13b4149330 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b414d840 con 0x7f13b40b0f50 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b414cfb0 con 0x7f13b4149330 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b414cc20 con 0x7f13b40b27c0 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7f13bac70640 10 monclient(hunting): authenticate will time out at 1538.490206s 2024-03-30T07:16:52.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7fc8b8d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7fc8b8d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.375+0000 7fc8b8d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.376+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.376+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1617636130 >> v2:172.21.15.47:3300/0 conn(0x7f82b00b0d20 msgr2=0x7f82b014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.376+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1617636130 >> v2:172.21.15.47:3300/0 conn(0x7f82b00b0d20 0x7f82b014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1617636130 >> v1:172.21.15.52:6789/0 conn(0x7f82b00b0120 legacy=0x7f82b00712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.377+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1617636130 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00b3780 con 0x7f82b00a84a0 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7fc8b3fff640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b406d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7fc8b3fff640 10 monclient(hunting): get_auth_request con 0x7fc8b406d290 auth_method 0 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7fc8b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7fc8b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7fc8b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.378+0000 7f13b920d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 1 -- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 msgr2=0x7fc8b406d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b406d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 1 -- 172.21.15.47:0/1449243680 >> v1:172.21.15.47:6789/0 conn(0x7fc8b40aed80 legacy=0x7fc8b40713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7fc8b8d6f640 1 -- 172.21.15.47:0/1449243680 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4071ae0 con 0x7fc8b4145bb0 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44154/0 (socket says 172.21.15.47:44154) 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 1 -- 172.21.15.47:0/2357812071 learned_addr learned my addr 172.21.15.47:0/2357812071 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 10 monclient(hunting): get_auth_request con 0x7f13b4149330 auth_method 0 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.379+0000 7f13b920d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.380+0000 7f13b920d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.380+0000 7f13b920d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.380+0000 7f13b920d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.381+0000 7fac68819640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.381+0000 7fac68819640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.381+0000 7fac68819640 10 monclient: monmap: 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.382151+0000 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.382151+0000 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.401 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.382+0000 7fac68819640 1 Processor -- start 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.382+0000 7fac68819640 1 -- start start 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.382+0000 7fac68819640 10 monclient: init 2024-03-30T07:16:52.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.382+0000 7fc8b8d6f640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b4145bb0 0x7fc8b4146060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009d90 tx=0x7fc8ac02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74cb0f252b740fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-03-30T07:16:52.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c038d80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac5c039160 0x7fac5c0c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 10 monclient(hunting): picked mon.noname-a con 0x7fac5c039160 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7fac68819640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac5c0c5770 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 -- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f13b40b27c0 msgr2=0x7f13b414c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f13b40b27c0 0x7f13b414c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 -- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b40b0f50 msgr2=0x7f13b4142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b40b0f50 0x7f13b4142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.383+0000 7f13b920d640 1 -- 172.21.15.47:0/2357812071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8009400 con 0x7f13b4149330 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c0c7e70 con 0x7fac5c0c5770 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c037610 con 0x7fac5c039160 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c035440 con 0x7fac5c038d80 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312803e5b0 con 0x7f3130144ab0 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312803e740 con 0x7f3130144ab0 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f3126ffd640 10 monclient: dump: 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.406 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7fac68819640 10 monclient(hunting): authenticate will time out at 1538.499323s 2024-03-30T07:16:52.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.384+0000 7f312cd99640 4 set_mon_vals no callback set 2024-03-30T07:16:52.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.385+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1617636130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 0x7f82b0144ab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009d90 tx=0x7f82a802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a6d1692b6167eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.386+0000 7f13b920d640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00cab0 tx=0x7f13ac00cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.386+0000 7f13b17fa640 10 client.?.objecter ms_handle_connect 0x7f13b4149330 2024-03-30T07:16:52.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.386+0000 7f13b17fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac00ac30 con 0x7f13b4149330 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 10 monclient: dump: 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.410 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ac00ad90 con 0x7f13b4149330 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.389+0000 7f13bac70640 5 monclient: authenticate success, global_id 4336 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 10 monclient: _renew_subs 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b414bdd0 con 0x7f13b4149330 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b2ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ac012600 con 0x7f13b4149330 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13b17fa640 10 monclient: dump: 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.412 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 10 monclient: _renew_subs 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b41407e0 con 0x7f13b4149330 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.390+0000 7f13bac70640 1 librados: init done 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 1 -- 172.21.15.47:0/1617636130 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a802cc90 con 0x7f82b00a84a0 2024-03-30T07:16:52.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: dump: 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.414 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 1 -- 172.21.15.47:0/1617636130 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a802cdf0 con 0x7f82b00a84a0 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 1 -- 172.21.15.47:0/1617636130 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8034660 con 0x7f82b00a84a0 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82adffb640 10 monclient: dump: 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.415 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7fac6259c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac5c039160 0x7fac5c0c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82b59b1640 5 monclient: authenticate success, global_id 4253 2024-03-30T07:16:52.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82b59b1640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.391+0000 7f82b59b1640 4 set_mon_vals no callback set 2024-03-30T07:16:52.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f13bac70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f13bac70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f13bac70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b414cde0 con 0x7f13b4149330 2024-03-30T07:16:52.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac5c039160 0x7fac5c0c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51060/0 (socket says 172.21.15.47:51060) 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 10 monclient: shutdown 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 -- 172.21.15.47:0/1617636130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 msgr2=0x7f82b0144ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1617636130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 0x7f82b0144ab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009d90 tx=0x7f82a802bee0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 -- 172.21.15.47:0/1617636130 shutdown_connections 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1617636130 >> v2:172.21.15.47:3300/0 conn(0x7f82b00b0d20 0x7f82b014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1617636130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00a84a0 0x7f82b0144ab0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 -- 172.21.15.47:0/1617636130 >> 172.21.15.47:0/1617636130 conn(0x7f82b00b70b0 msgr2=0x7f82b00b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 1 -- 172.21.15.47:0/93445013 learned_addr learned my addr 172.21.15.47:0/93445013 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 10 monclient(hunting): get_auth_request con 0x7fac5c039160 auth_method 0 2024-03-30T07:16:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7fac6259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.392+0000 7f82b59b1640 1 -- 172.21.15.47:0/1617636130 shutdown_connections 2024-03-30T07:16:52.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7f82b59b1640 1 -- 172.21.15.47:0/1617636130 wait complete. 2024-03-30T07:16:52.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fac62d9d640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 1 -- 172.21.15.47:0/1449243680 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac02cab0 con 0x7fc8b4145bb0 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: dump: 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.424 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 1 -- 172.21.15.47:0/1449243680 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac02cc10 con 0x7fc8b4145bb0 2024-03-30T07:16:52.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 1 -- 172.21.15.47:0/1449243680 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac034440 con 0x7fc8b4145bb0 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b2ffd640 10 monclient: dump: 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.426 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b9570640 5 monclient: authenticate success, global_id 4308 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b9570640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.393+0000 7fc8b9570640 4 set_mon_vals no callback set 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 10 monclient: shutdown 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 -- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b4145bb0 msgr2=0x7fc8b4146060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b4145bb0 0x7fc8b4146060 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009d90 tx=0x7fc8ac02bd00 comp rx=0 tx=0).stop 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 -- 172.21.15.47:0/1449243680 shutdown_connections 2024-03-30T07:16:52.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b4145bb0 0x7fc8b4146060 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1449243680 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b406d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7fc8b9570640 1 -- 172.21.15.47:0/1449243680 >> 172.21.15.47:0/1449243680 conn(0x7fc8b40af610 msgr2=0x7fc8b40afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac62d9d640 10 monclient(hunting): get_auth_request con 0x7fac5c038d80 auth_method 0 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac62d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fac62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 10 monclient: monmap: 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 librados: starting msgr at 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 librados: starting objecter 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 Processor -- start 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 -- start start 2024-03-30T07:16:52.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 librados: setting wanted keys 2024-03-30T07:16:52.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 1 librados: calling monclient init 2024-03-30T07:16:52.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.394+0000 7f82b59b1640 10 monclient: init 2024-03-30T07:16:52.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fc8b9570640 1 -- 172.21.15.47:0/1449243680 shutdown_connections 2024-03-30T07:16:52.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7fc8b9570640 1 -- 172.21.15.47:0/1449243680 wait complete. 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7f82b59b1640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7f82b59b1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7f82b59b1640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82b00a84a0 0x7f82b01d9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-b con 0x7f82b00a84a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.395+0000 7f82b59b1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-a con 0x7f82b00b0120 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00b0d20 0x7f82b013bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7fac62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7fac62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7fac62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): picked mon.noname-c con 0x7f82b00b0d20 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00af6d0 con 0x7f82b00b0120 2024-03-30T07:16:52.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b01da690 con 0x7f82b00a84a0 2024-03-30T07:16:52.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b013c870 con 0x7f82b00b0d20 2024-03-30T07:16:52.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7f82b59b1640 10 monclient(hunting): authenticate will time out at 1538.511312s 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7fc8b9570640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.396+0000 7fc8b9570640 10 monclient: monmap: 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.435 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 librados: starting msgr at 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 librados: starting objecter 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 Processor -- start 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 -- start start 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 librados: setting wanted keys 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 1 librados: calling monclient init 2024-03-30T07:16:52.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.397+0000 7fc8b9570640 10 monclient: init 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 1 -- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 msgr2=0x7fac5c0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c0c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 1 -- 172.21.15.47:0/93445013 >> v1:172.21.15.47:6789/0 conn(0x7fac5c0c5770 legacy=0x7fac5c0cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.398+0000 7fac6259c640 1 -- 172.21.15.47:0/93445013 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c035870 con 0x7fac5c039160 2024-03-30T07:16:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b40abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b406d290 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b40aed80 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8b4145bb0 0x7fc8b40a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b4145bb0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40a9e80 con 0x7fc8b40aed80 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4147340 con 0x7fc8b4145bb0 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4148580 con 0x7fc8b406d290 2024-03-30T07:16:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.399+0000 7fc8b9570640 10 monclient(hunting): authenticate will time out at 1538.514759s 2024-03-30T07:16:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.405+0000 7fac6259c640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c039160 0x7fac5c0c8320 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac580098a0 tx=0x7fac5802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac80f6d55b99585a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.406+0000 7fc8b37fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.406+0000 7f82ae7fc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51070/0 (socket says 172.21.15.47:51070) 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1865941726 learned_addr learned my addr 172.21.15.47:0/1865941726 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 10 monclient(hunting): get_auth_request con 0x7f82b00b0120 auth_method 0 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.407+0000 7f82ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51086/0 (socket says 172.21.15.47:51086) 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 1 -- 172.21.15.47:0/1868753564 learned_addr learned my addr 172.21.15.47:0/1868753564 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 10 monclient(hunting): get_auth_request con 0x7fc8b40aed80 auth_method 0 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7fc8b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.409+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f13ac012760 con 0x7f13b4149330 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f1038e69640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f1038e69640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f1038e69640 10 monclient: monmap: 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.411269+0000 2024-03-30T07:16:52.444 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.411269+0000 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f13b17fa640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 0x7f139806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f13b8a0c640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 0x7f139806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f13b8a0c640 10 monclient: get_auth_request con 0x7f139806c590 auth_method 0 2024-03-30T07:16:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.410+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.411+0000 7f13b17fa640 10 client.4336.objecter ms_handle_connect 0x7f139806c590 2024-03-30T07:16:52.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.411+0000 7f13b8a0c640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 0x7f139806ea50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f13b40b2510 tx=0x7f13a8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.411+0000 7f1038e69640 1 Processor -- start 2024-03-30T07:16:52.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.411+0000 7f1038e69640 1 -- start start 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.411+0000 7f1038e69640 10 monclient: init 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f10340b0120 0x7f10340a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-a con 0x7f10340b0120 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1034144ab0 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10340b0d20 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10340a8bd0 con 0x7f10340b0d20 2024-03-30T07:16:52.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1034144f10 con 0x7f1034144ab0 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10340aa4f0 con 0x7f10340b0120 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.412+0000 7f1038e69640 10 monclient(hunting): authenticate will time out at 1538.527417s 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00b0d20 msgr2=0x7f82b013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00b0d20 0x7f82b013bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82b00a84a0 msgr2=0x7f82b01d9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82b00a84a0 0x7f82b01d9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.415+0000 7f82ae7fc640 1 -- 172.21.15.47:0/1865941726 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8009a70 con 0x7f82b00b0120 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.416+0000 7fc8b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.416+0000 7fc8b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.416+0000 7fc8b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 -- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 msgr2=0x7fc8b40abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b40abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 -- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8b4145bb0 msgr2=0x7fc8b40a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8b4145bb0 0x7fc8b40a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 monclient: found mon.noname-a 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.420+0000 7fc8b37fe640 1 -- 172.21.15.47:0/1868753564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac009a70 con 0x7fc8b40aed80 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 1 -- 172.21.15.47:0/93445013 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580317c0 con 0x7fac5c039160 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: dump: 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.454 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 1 -- 172.21.15.47:0/93445013 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58031920 con 0x7fac5c039160 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac68819640 5 monclient: authenticate success, global_id 4314 2024-03-30T07:16:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac68819640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac68819640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 1 -- 172.21.15.47:0/93445013 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58031cb0 con 0x7fac5c039160 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac68819640 4 set_mon_vals no callback set 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.423+0000 7fac61d9b640 10 monclient: dump: 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.456 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 10 monclient: shutdown 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 -- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c039160 msgr2=0x7fac5c0c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c039160 0x7fac5c0c8320 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fac580098a0 tx=0x7fac5802b860 comp rx=0 tx=0).stop 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 -- 172.21.15.47:0/93445013 shutdown_connections 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c039160 0x7fac5c0c8320 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 --2- 172.21.15.47:0/93445013 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c0c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 -- 172.21.15.47:0/93445013 >> 172.21.15.47:0/93445013 conn(0x7fac5c0065e0 msgr2=0x7fac5c0069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 -- 172.21.15.47:0/93445013 shutdown_connections 2024-03-30T07:16:52.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.424+0000 7fac68819640 1 -- 172.21.15.47:0/93445013 wait complete. 2024-03-30T07:16:52.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.425+0000 7f82ae7fc640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009d90 tx=0x7f82a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.425+0000 7f8297fff640 10 client.?.objecter ms_handle_connect 0x7f82b00b0120 2024-03-30T07:16:52.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.425+0000 7f8297fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.425+0000 7fac68819640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.425+0000 7fac68819640 10 monclient: monmap: 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 librados: starting msgr at 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 librados: starting objecter 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 Processor -- start 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fc8b37fe640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c430 tx=0x7fc8a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fc8b0ff9640 10 client.?.objecter ms_handle_connect 0x7fc8b40aed80 2024-03-30T07:16:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fc8b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 -- start start 2024-03-30T07:16:52.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 librados: setting wanted keys 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 1 librados: calling monclient init 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.426+0000 7fac68819640 10 monclient: init 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c02fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): picked mon.noname-c con 0x7fac5c038d80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c0c5770 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c030560 0x7fac5c15fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): picked mon.noname-a con 0x7fac5c030560 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0cbd60 con 0x7fac5c030560 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0cc1d0 con 0x7fac5c0c5770 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c0cbef0 con 0x7fac5c038d80 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.427+0000 7fac68819640 10 monclient(hunting): authenticate will time out at 1538.542551s 2024-03-30T07:16:52.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8005ce0 con 0x7fc8b40aed80 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 10 monclient: dump: 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.464 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 5 monclient: authenticate success, global_id 4320 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 10 monclient: _renew_subs 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b40ab190 con 0x7fc8b40aed80 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 10 monclient: _renew_subs 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b40ac560 con 0x7fc8b40aed80 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.432+0000 7fc8b9570640 1 librados: init done 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.433+0000 7fc8b9570640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.433+0000 7fc8b9570640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.433+0000 7fc8b9570640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.433+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b4147060 con 0x7fc8b40aed80 2024-03-30T07:16:52.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.434+0000 7f103259c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f10340b0120 0x7f10340a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.439+0000 7fa91ea22640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.439+0000 7fa91ea22640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.439+0000 7fa91ea22640 10 monclient: monmap: 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.439898+0000 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.439898+0000 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.467 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.468 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.468 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.468 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.468 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7fa91ea22640 1 Processor -- start 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f10340b0120 0x7f10340a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51098/0 (socket says 172.21.15.47:51098) 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 1 -- 172.21.15.47:0/1961743902 learned_addr learned my addr 172.21.15.47:0/1961743902 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 10 monclient(hunting): get_auth_request con 0x7f10340b0120 auth_method 0 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7f103259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7fa91ea22640 1 -- start start 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.440+0000 7fa91ea22640 10 monclient: init 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa910037570 0x7fa91002b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-a con 0x7fa910037570 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa91002a8f0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-b con 0x7fa910033e70 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91002c540 con 0x7fa91002a8f0 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa910037950 con 0x7fa910037570 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91002ae60 con 0x7fa910033e70 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.441+0000 7fa91ea22640 10 monclient(hunting): authenticate will time out at 1538.556656s 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a803e610 con 0x7f82b00b0120 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: dump: 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a803e790 con 0x7f82b00b0120 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a803eb40 con 0x7f82b00b0120 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7f8297fff640 10 monclient: dump: 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.473 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.442+0000 7fac6359e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c030560 0x7fac5c15fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.443+0000 7f82acff9640 4 set_mon_vals no callback set 2024-03-30T07:16:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 5 monclient: authenticate success, global_id 4317 2024-03-30T07:16:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 10 monclient: _renew_subs 2024-03-30T07:16:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b013ca90 con 0x7f82b00b0120 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 10 monclient: _renew_subs 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b013c280 con 0x7f82b00b0120 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.445+0000 7f82b59b1640 1 librados: init done 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.446+0000 7f82b59b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.446+0000 7f82b59b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.446+0000 7f82b59b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:52.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.446+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b004e030 con 0x7f82b00b0120 2024-03-30T07:16:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 10 monclient: get_monmap_and_config 2024-03-30T07:16:52.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 10 monclient: monmap: 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:52.478880+0000 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:52.478880+0000 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:52.479 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 1 Processor -- start 2024-03-30T07:16:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 1 -- start start 2024-03-30T07:16:52.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.478+0000 7fb8ccd2a640 10 monclient: init 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8c80b0120 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8c80a84a0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8c814ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c8144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80a8900 con 0x7fb8c80a84a0 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80a8bd0 con 0x7fb8c80b0120 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80b1430 con 0x7fb8c8144ab0 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.479+0000 7fb8ccd2a640 10 monclient(hunting): authenticate will time out at 1538.594975s 2024-03-30T07:16:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.669+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f13ac058910 con 0x7f13b4149330 2024-03-30T07:16:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.669+0000 7f13b17fa640 10 client.4336.objecter ms_dispatch 0x7f13b4000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.669+0000 7f13b17fa640 3 client.4336.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.669+0000 7f13b17fa640 3 client.4336.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.669+0000 7f13b17fa640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.674+0000 7fac62d9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c02fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.679+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f13ac054c50 con 0x7f13b4149330 2024-03-30T07:16:52.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.679+0000 7f13b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.679+0000 7f13b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.682+0000 7fac6259c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44176/0 (socket says 172.21.15.47:44176) 2024-03-30T07:16:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 1 -- 172.21.15.47:0/3502651278 learned_addr learned my addr 172.21.15.47:0/3502651278 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 10 monclient(hunting): get_auth_request con 0x7fac5c0c5770 auth_method 0 2024-03-30T07:16:52.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.683+0000 7fac6259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.684+0000 7fb8c659c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb8c80b0120 0x7fb8c80712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56282/0 (socket says 172.21.15.47:56282) 2024-03-30T07:16:52.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.684+0000 7fb8c659c640 1 -- 172.21.15.47:0/1256512681 learned_addr learned my addr 172.21.15.47:0/1256512681 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:52.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.684+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.684+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.684+0000 7fac6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 -- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 msgr2=0x7fac5c02fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c02fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 -- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c030560 msgr2=0x7fac5c15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c030560 0x7fac5c15fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.688+0000 7fac6259c640 1 -- 172.21.15.47:0/3502651278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580094d0 con 0x7fac5c0c5770 2024-03-30T07:16:52.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.689+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f312803e8a0 con 0x7f3130144ab0 2024-03-30T07:16:52.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.690+0000 7f3126ffd640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 0x7f311406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.690+0000 7f312e59c640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 0x7f311406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.690+0000 7f312e59c640 10 monclient: get_auth_request con 0x7f311406c590 auth_method 0 2024-03-30T07:16:52.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.691+0000 7f312e59c640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 0x7f311406ea50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3118006fd0 tx=0x7f3118008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.691+0000 7f3126ffd640 10 client.4250.objecter ms_handle_connect 0x7f311406c590 2024-03-30T07:16:52.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.691+0000 7fac6259c640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac580090b0 tx=0x7fac580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.691+0000 7fac4b7fe640 10 client.?.objecter ms_handle_connect 0x7fac5c0c5770 2024-03-30T07:16:52.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.691+0000 7fac4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303705706 0 0) 0x7f1034144f10 con 0x7f1034144ab0 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 10 monclient(hunting): _negotiate 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 10 monclient(hunting): my global_id is 4342 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.693+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c003520 con 0x7f1034144ab0 2024-03-30T07:16:52.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.694+0000 7fa91e221640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3941725283 0 0) 0x7f101c003520 con 0x7f1034144ab0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 >> v2:172.21.15.47:3300/0 conn(0x7f10340b0120 msgr2=0x7f10340a84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 --2- 172.21.15.47:0/1961743902 >> v2:172.21.15.47:3300/0 conn(0x7f10340b0120 0x7f10340a84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 >> v1:172.21.15.47:6789/0 conn(0x7f10340b0d20 legacy=0x7f103414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 monclient: found mon.noname-c-legacy 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.52:6789/0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10340b3780 con 0x7f1034144ab0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f103159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f1038e69640 5 monclient: authenticate success, global_id 4342 2024-03-30T07:16:52.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.697+0000 7f1038e69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f1028001670 con 0x7f1034144ab0 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient: dump: 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.702 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] but i'm connected to v1:172.21.15.52:6789/0, reconnecting 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 >> v1:172.21.15.52:6789/0 conn(0x7f1034144ab0 legacy=0x7f10340712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 0x7f101c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): picked mon.b con 0x7f101c003970 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c0047c0 0x7f101c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): picked mon.a con 0x7f101c0047c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): picked mon.c con 0x7f101c005180 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10340aa4f0 con 0x7f101c0047c0 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c00c0b0 con 0x7f101c003970 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c00c270 con 0x7f101c005180 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f103159a640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:16:52.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.700+0000 7f1038e69640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856531497 0 0) 0x7fb8c80a8bd0 con 0x7fb8c80b0120 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 10 monclient(hunting): _negotiate 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 10 monclient(hunting): my global_id is 4345 2024-03-30T07:16:52.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.701+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b0003520 con 0x7fb8c80b0120 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f312807b170 con 0x7f3130144ab0 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 10 client.4250.objecter ms_dispatch 0x7f3130000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 3 client.4250.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 3 client.4250.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f31280774b0 con 0x7f3130144ab0 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f3126ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580329a0 con 0x7fac5c0c5770 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: dump: 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.707 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58032b40 con 0x7fac5c0c5770 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580453f0 con 0x7fac5c0c5770 2024-03-30T07:16:52.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7f103259c640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 0x7f101c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:52.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac4b7fe640 10 monclient: dump: 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 5 monclient: authenticate success, global_id 4339 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 10 monclient: _renew_subs 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c0cb190 con 0x7fac5c0c5770 2024-03-30T07:16:52.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac6159a640 4 set_mon_vals no callback set 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 10 monclient: _renew_subs 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c15d000 con 0x7fac5c0c5770 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.705+0000 7fac68819640 1 librados: init done 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fac68819640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fac68819640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fac68819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c030280 con 0x7fac5c0c5770 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.1 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698768368 0 0) 0x7fb8b0003520 con 0x7fb8c80b0120 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 msgr2=0x7fb8c814ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8c814ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 >> v1:172.21.15.47:6789/0 conn(0x7fb8c80a84a0 legacy=0x7fb8c80b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 monclient: found mon.noname-c-legacy 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.52:6789/0 2024-03-30T07:16:52.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8144f40 con 0x7fb8c80b0120 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.1 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7fb8b8001670 con 0x7fb8c80b0120 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7f103259c640 10 monclient(hunting): get_auth_request con 0x7f101c003970 auth_method 0 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7f103259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7f103259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7f103259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fa91e221640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54902/0 (socket says 172.21.15.47:54902) 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fa91e221640 1 -- 172.21.15.47:0/1712806380 learned_addr learned my addr 172.21.15.47:0/1712806380 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fa91e221640 10 monclient(hunting): get_auth_request con 0x7fa910033e70 auth_method 0 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.706+0000 7fb8c559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient: dump: 2024-03-30T07:16:52.714 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] but i'm connected to v1:172.21.15.52:6789/0, reconnecting 2024-03-30T07:16:52.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 >> v1:172.21.15.52:6789/0 conn(0x7fb8c80b0120 legacy=0x7fb8c80712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8b0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): picked mon.c con 0x7fb8c8144ab0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fa91e221640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fa91e221640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fa91e221640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8b00040b0 0x7fb8b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): picked mon.a con 0x7fb8b00040b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8b0004e20 0x7fb8b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): picked mon.b con 0x7fb8b0004e20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80b1430 con 0x7fb8b00040b0 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b0005500 con 0x7fb8b0004e20 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b000bec0 con 0x7fb8c8144ab0 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.707+0000 7fb8c559a640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.708+0000 7f103259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.708+0000 7f103259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.708+0000 7f103259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.709+0000 7fa91e221640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.709+0000 7fa91e221640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.709+0000 7fa91e221640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 10 monclient(hunting): handle_auth_done global_id 4342 payload 293 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 -- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f101c005180 msgr2=0x7f101c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 -- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c0047c0 msgr2=0x7f101c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c0047c0 0x7f101c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 monclient: found mon.b 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 1 -- 172.21.15.47:0/1961743902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c003520 con 0x7f101c003970 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f1032d9d640 1 -- 172.21.15.47:0/1961743902 reap_dead start 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.710+0000 7f103259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 1 -- 172.21.15.47:0/1712806380 >> v2:172.21.15.47:3300/0 conn(0x7fa910037570 msgr2=0x7fa91002b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 1 --2- 172.21.15.47:0/1712806380 >> v2:172.21.15.47:3300/0 conn(0x7fa910037570 0x7fa91002b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 1 -- 172.21.15.47:0/1712806380 >> v1:172.21.15.47:6789/0 conn(0x7fa91002a8f0 legacy=0x7fa9100cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 1 monclient: found mon.noname-b 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.713+0000 7fa91e221640 1 -- 172.21.15.47:0/1712806380 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100cc720 con 0x7fa910033e70 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.715+0000 7fa91e221640 1 --2- 172.21.15.47:0/1712806380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa918069740 tx=0x7fa9180a5310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e9060af773684b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.718+0000 7fb8c659c640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8b0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.720+0000 7fb8c659c640 10 monclient(hunting): get_auth_request con 0x7fb8c8144ab0 auth_method 0 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.720+0000 7fb8c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.720+0000 7fb8c659c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.720+0000 7fb8c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.721+0000 7fb8c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.721+0000 7fb8c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.721+0000 7fb8c659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 10 monclient(hunting): handle_auth_done global_id 4345 payload 293 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 -- 172.21.15.47:0/1256512681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8b0004e20 msgr2=0x7fb8b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8b0004e20 0x7fb8b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 -- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8b00040b0 msgr2=0x7fb8b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8b00040b0 0x7fb8b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 monclient: found mon.c 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 1 -- 172.21.15.47:0/1256512681 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0003520 con 0x7fb8c8144ab0 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.725+0000 7fb8c659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.726+0000 7fb8ccd2a640 5 monclient: authenticate success, global_id 4345 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.726+0000 7fb8ccd2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 1 -- 172.21.15.47:0/1712806380 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9180ad640 con 0x7fa910033e70 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: dump: 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.728 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 1 -- 172.21.15.47:0/1712806380 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9180ad7a0 con 0x7fa910033e70 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 1 -- 172.21.15.47:0/1712806380 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9180adbd0 con 0x7fa910033e70 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ca1e640 10 monclient: dump: 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ea22640 5 monclient: authenticate success, global_id 4262 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ea22640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.727+0000 7fa91ea22640 4 set_mon_vals no callback set 2024-03-30T07:16:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 10 monclient: shutdown 2024-03-30T07:16:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 -- 172.21.15.47:0/1712806380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 msgr2=0x7fa9100c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 --2- 172.21.15.47:0/1712806380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa918069740 tx=0x7fa9180a5310 comp rx=0 tx=0).stop 2024-03-30T07:16:52.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 -- 172.21.15.47:0/1712806380 shutdown_connections 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 --2- 172.21.15.47:0/1712806380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100c92f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 --2- 172.21.15.47:0/1712806380 >> v2:172.21.15.47:3300/0 conn(0x7fa910037570 0x7fa91002b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 -- 172.21.15.47:0/1712806380 >> 172.21.15.47:0/1712806380 conn(0x7fa91003a8e0 msgr2=0x7fa910036b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 -- 172.21.15.47:0/1712806380 shutdown_connections 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.728+0000 7fa91ea22640 1 -- 172.21.15.47:0/1712806380 wait complete. 2024-03-30T07:16:52.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.729+0000 7fb8c659c640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8b0003970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800c0e0 tx=0x7fb8b00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e86b74c62f0597 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.729+0000 7fa91ea22640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.729+0000 7fa91ea22640 10 monclient: monmap: 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 librados: starting msgr at 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 librados: starting objecter 2024-03-30T07:16:52.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 Processor -- start 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 -- start start 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 librados: setting wanted keys 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 1 librados: calling monclient init 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.730+0000 7fa91ea22640 10 monclient: init 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa91002a8f0 0x7fa9100c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-a con 0x7fa91002a8f0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-c con 0x7fa910033e70 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa910037570 0x7fa9100bc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): picked mon.noname-b con 0x7fa910037570 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9100c46b0 con 0x7fa91002a8f0 2024-03-30T07:16:52.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9100d63e0 con 0x7fa910037570 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9100c48b0 con 0x7fa910033e70 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.731+0000 7fa91ea22640 10 monclient(hunting): authenticate will time out at 1538.846839s 2024-03-30T07:16:52.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.736+0000 7fa91d21f640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8005ad0 con 0x7fb8c8144ab0 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: dump: 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.738 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b8005c30 con 0x7fb8c8144ab0 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 1 -- 172.21.15.47:0/1256512681 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b802c7d0 con 0x7fb8c8144ab0 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8c559a640 10 monclient: dump: 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.739 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8ccd2a640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:52.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.737+0000 7fb8ccd2a640 4 set_mon_vals no callback set 2024-03-30T07:16:52.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54950/0 (socket says 172.21.15.47:54950) 2024-03-30T07:16:52.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 1 -- 172.21.15.47:0/657022570 learned_addr learned my addr 172.21.15.47:0/657022570 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 10 monclient(hunting): get_auth_request con 0x7fa910033e70 auth_method 0 2024-03-30T07:16:52.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fa91d21f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 10 monclient: shutdown 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 msgr2=0x7fb8b0003970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/1256512681 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8144ab0 0x7fb8b0003970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800c0e0 tx=0x7fb8b00057f0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8c6d9d640 1 -- 172.21.15.47:0/1256512681 reap_dead start 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/1256512681 shutdown_connections 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.738+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/1256512681 >> 172.21.15.47:0/1256512681 conn(0x7fb8c80b70b0 msgr2=0x7fb8c80b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.739+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/1256512681 shutdown_connections 2024-03-30T07:16:52.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.739+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/1256512681 wait complete. 2024-03-30T07:16:52.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.739+0000 7fa91d21f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.739+0000 7fa91d21f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.739+0000 7fa91d21f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.740+0000 7fb8ccd2a640 10 monclient: build_initial_monmap 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.740+0000 7fb8ccd2a640 10 monclient: monmap: 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.740+0000 7fb8ccd2a640 1 librados: starting msgr at 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fb8ccd2a640 1 librados: starting objecter 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fb8ccd2a640 1 Processor -- start 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fb8ccd2a640 1 -- start start 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a8005e40 con 0x7fc8b40aed80 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8004b40 con 0x7fc8b40aed80 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b0ff9640 10 monclient: dump: 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.747 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fc8b27fc640 4 set_mon_vals no callback set 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.741+0000 7fb8ccd2a640 1 librados: setting wanted keys 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.742+0000 7fb8ccd2a640 1 librados: calling monclient init 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.742+0000 7fb8ccd2a640 10 monclient: init 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.742+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc8a8007330 con 0x7fc8b40aed80 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.742+0000 7fb8ccd2a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:52.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.742+0000 7fb8ccd2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:52.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fc8b0ff9640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 0x7fc8a006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8144ab0 0x7fb8c80ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c8144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c8140500 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8c81d6bf0 0x7fb8c81d7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c81d6bf0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fc8b3fff640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 0x7fc8a006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fc8b3fff640 10 monclient: get_auth_request con 0x7fc8a006c590 auth_method 0 2024-03-30T07:16:52.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c8139e30 con 0x7fb8c8144ab0 2024-03-30T07:16:52.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80ac630 con 0x7fb8c81d6bf0 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c80ad750 con 0x7fb8c8140500 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.743+0000 7fb8ccd2a640 10 monclient(hunting): authenticate will time out at 1538.858207s 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fc8b3fff640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 0x7fc8a006ea50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc8a40098f0 tx=0x7fc8a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fc8b0ff9640 10 client.4320.objecter ms_handle_connect 0x7fc8a006c590 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 -- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa910037570 msgr2=0x7fa9100bc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa910037570 0x7fa9100bc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 -- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa91002a8f0 msgr2=0x7fa9100c10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa91002a8f0 0x7fa9100c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.744+0000 7fa91d21f640 1 -- 172.21.15.47:0/657022570 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91806be70 con 0x7fa910033e70 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54966/0 (socket says 172.21.15.47:54966) 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 1 -- 172.21.15.47:0/3415949614 learned_addr learned my addr 172.21.15.47:0/3415949614 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb8c8140500 auth_method 0 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:52.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.747+0000 7fb8c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.748+0000 7fa91d21f640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c440 tx=0x7fa90800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.748+0000 7fa90e7fc640 10 client.?.objecter ms_handle_connect 0x7fa910033e70 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.748+0000 7fa90e7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.749+0000 7fb8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.749+0000 7fb8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.749+0000 7fb8c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-03-30T07:16:52.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 -- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8c81d6bf0 msgr2=0x7fb8c81d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8c81d6bf0 0x7fb8c81d7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 -- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8144ab0 msgr2=0x7fb8c80ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8144ab0 0x7fb8c80ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 monclient: found mon.noname-c 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.756+0000 7fb8c5d9b640 1 -- 172.21.15.47:0/3415949614 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c81421f0 con 0x7fb8c8140500 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908005480 con 0x7fa910033e70 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: dump: 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.761 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9080035f0 con 0x7fa910033e70 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9080039c0 con 0x7fa910033e70 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90e7fc640 10 monclient: dump: 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.762 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa90ffff640 4 set_mon_vals no callback set 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa91ea22640 5 monclient: authenticate success, global_id 4265 2024-03-30T07:16:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa91ea22640 10 monclient: _renew_subs 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa91ea22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.760+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9100c0260 con 0x7fa910033e70 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fa91ea22640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fa91ea22640 10 monclient: _renew_subs 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fa91ea22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9100c1b80 con 0x7fa910033e70 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fa91ea22640 1 librados: init done 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fb8c5d9b640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c440 tx=0x7fb8bc00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fb8b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb8c8140500 2024-03-30T07:16:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.761+0000 7fb8b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:52.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.762+0000 7fa91ea22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.762+0000 7fa91ea22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.762+0000 7fa91ea22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.762+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9100d65a0 con 0x7fa910033e70 2024-03-30T07:16:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc005480 con 0x7fb8c8140500 2024-03-30T07:16:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: dump: 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8bc0035f0 con 0x7fb8c8140500 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0039a0 con 0x7fb8c8140500 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8b6ffd640 10 monclient: dump: 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:52.768 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 5 monclient: authenticate success, global_id 4268 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 10 monclient: _renew_subs 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c81d66d0 con 0x7fb8c8140500 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 10 monclient: _renew_subs 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c80af170 con 0x7fb8c8140500 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8ccd2a640 1 librados: init done 2024-03-30T07:16:52.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc8a8057ec0 con 0x7fc8b40aed80 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 10 client.4320.objecter ms_dispatch 0x7fc8b4000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 3 client.4320.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 3 client.4320.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fb8c4d99640 4 set_mon_vals no callback set 2024-03-30T07:16:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.766+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc8a8054200 con 0x7fc8b40aed80 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fc8b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fc8b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fb8ccd2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fb8ccd2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fb8ccd2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.767+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac005040 con 0x7fb8c8140500 2024-03-30T07:16:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.812+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa908003b20 con 0x7fa910033e70 2024-03-30T07:16:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.813+0000 7fa90e7fc640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 0x7fa8fc06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.813+0000 7fa91da20640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 0x7fa8fc06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.813+0000 7fa91da20640 10 monclient: get_auth_request con 0x7fa8fc06c590 auth_method 0 2024-03-30T07:16:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.814+0000 7fa91da20640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 0x7fa8fc06ea50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa914009670 tx=0x7fa914006bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.814+0000 7fa90e7fc640 10 client.4265.objecter ms_handle_connect 0x7fa8fc06c590 2024-03-30T07:16:52.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.818+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa908057d50 con 0x7fa910033e70 2024-03-30T07:16:52.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.818+0000 7fa90e7fc640 10 client.4265.objecter ms_dispatch 0x7fa918000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.818+0000 7fa90e7fc640 3 client.4265.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.818+0000 7fa90e7fc640 3 client.4265.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.818+0000 7fa90e7fc640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-30T07:16:52.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.894+0000 7f13bac70640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:52.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.894+0000 7f13bac70640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:52.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.894+0000 7f13bac70640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:52.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.894+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f13b404e030 con 0x7f13b4149330 2024-03-30T07:16:52.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.970+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb8bc003b00 con 0x7fb8c8140500 2024-03-30T07:16:52.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.970+0000 7fb8b6ffd640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 0x7fb8a806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.970+0000 7fb8c659c640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 0x7fb8a806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.971+0000 7fb8c659c640 10 monclient: get_auth_request con 0x7fb8a806c610 auth_method 0 2024-03-30T07:16:52.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.971+0000 7fb8b6ffd640 10 client.4268.objecter ms_handle_connect 0x7fb8a806c610 2024-03-30T07:16:52.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.971+0000 7fb8c659c640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 0x7fb8a806ead0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8034000 tx=0x7fb8b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.984+0000 7f31349ab640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.984+0000 7f31349ab640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.984+0000 7f31349ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.984+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f313004e030 con 0x7f3130144ab0 2024-03-30T07:16:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.986+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fac5804e450 con 0x7fac5c0c5770 2024-03-30T07:16:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.987+0000 7fac4b7fe640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 0x7fac5006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.987+0000 7fac62d9d640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 0x7fac5006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.987+0000 7fac62d9d640 10 monclient: get_auth_request con 0x7fac5006c5c0 auth_method 0 2024-03-30T07:16:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.987+0000 7fac4b7fe640 10 client.4339.objecter ms_handle_connect 0x7fac5006c5c0 2024-03-30T07:16:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.987+0000 7fac62d9d640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 0x7fac5006ea80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac54009a10 tx=0x7fac54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:52.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.990+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fac5807a620 con 0x7fac5c0c5770 2024-03-30T07:16:52.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.990+0000 7fac4b7fe640 10 client.4339.objecter ms_dispatch 0x7fac64000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:52.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.990+0000 7fac4b7fe640 3 client.4339.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:52.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.990+0000 7fac4b7fe640 3 client.4339.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:52.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:52.990+0000 7fac4b7fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-30T07:16:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.002+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac58076960 con 0x7fac5c0c5770 2024-03-30T07:16:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.002+0000 7fac4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.002+0000 7fac4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:53.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.005+0000 7f103259c640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 0x7f101c003d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10240098a0 tx=0x7f101c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36f74147e9e1727e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024030660 con 0x7f101c003970 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: dump: 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:53.019 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f1038e69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10240307c0 con 0x7f101c003970 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f1038e69640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 1 -- 172.21.15.47:0/1961743902 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024030b50 con 0x7f101c003970 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f1038e69640 4 set_mon_vals no callback set 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.017+0000 7f103159a640 10 monclient: dump: 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.020 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 10 monclient: shutdown 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 -- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 msgr2=0x7f101c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 0x7f101c003d70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f10240098a0 tx=0x7f101c0057f0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 -- 172.21.15.47:0/1961743902 shutdown_connections 2024-03-30T07:16:53.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 --2- 172.21.15.47:0/1961743902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f101c003970 0x7f101c003d70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 -- 172.21.15.47:0/1961743902 >> 172.21.15.47:0/1961743902 conn(0x7f10340b70b0 msgr2=0x7f10340b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 -- 172.21.15.47:0/1961743902 shutdown_connections 2024-03-30T07:16:53.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.018+0000 7f1038e69640 1 -- 172.21.15.47:0/1961743902 wait complete. 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f1038e69640 10 monclient: build_initial_monmap 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f1038e69640 10 monclient: monmap: 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:53.023 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:53.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f13b17fa640 1 -- 172.21.15.47:0/2357812071 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13ac021090 con 0x7f13b4149330 2024-03-30T07:16:53.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f13b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:53.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f13b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.024 INFO:teuthology.orchestra.run.smithi047.stdout:60129542148 2024-03-30T07:16:53.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f1038e69640 1 librados: starting msgr at 2024-03-30T07:16:53.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.019+0000 7f1038e69640 1 librados: starting objecter 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 1 Processor -- start 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 1 -- start start 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 1 librados: setting wanted keys 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 1 librados: calling monclient init 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 10 monclient: init 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.020+0000 7f1038e69640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c003970 0x7f1034141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-a con 0x7f101c003970 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:53.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-b con 0x7f10341d6600 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:53.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f10341d99e0 0x7f10341d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:53.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): picked mon.noname-c con 0x7f10341d99e0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:53.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:53.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10340ae7b0 con 0x7f101c003970 2024-03-30T07:16:53.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10340afe40 con 0x7f10341d6600 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10341da380 con 0x7f10341d99e0 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1038e69640 10 monclient(hunting): authenticate will time out at 1539.136619s 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.021+0000 7f1031d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 10 librados: watch_flush enter 2024-03-30T07:16:53.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 10 librados: watch_flush exit 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 10 client.4336.objecter successfully canceled tick 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44244/0 (socket says 172.21.15.47:44244) 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 1 -- 172.21.15.47:0/3963426815 learned_addr learned my addr 172.21.15.47:0/3963426815 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): get_auth_request con 0x7f10341d6600 auth_method 0 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:53.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 msgr2=0x7f139806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 0x7f139806ea50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f13b40b2510 tx=0x7f13a8030000 comp rx=0 tx=0).stop 2024-03-30T07:16:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 10 monclient: shutdown 2024-03-30T07:16:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 msgr2=0x7f13b41400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f13ac00cab0 tx=0x7f13ac00cf70 comp rx=0 tx=0).stop 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 shutdown_connections 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f139806c590 0x7f139806ea50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f13b4149330 0x7f13b41400c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f13b40b27c0 0x7f13b414c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 --2- 172.21.15.47:0/2357812071 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f13b40b0f50 0x7f13b4142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 >> 172.21.15.47:0/2357812071 conn(0x7f13b40b70b0 msgr2=0x7f13b41497a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 shutdown_connections 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 -- 172.21.15.47:0/2357812071 wait complete. 2024-03-30T07:16:53.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f13bac70640 1 librados: shutdown 2024-03-30T07:16:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.022+0000 7f1031d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.024+0000 7fc8b9570640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:53.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.024+0000 7fc8b9570640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:53.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.024+0000 7fc8b9570640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:53.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.024+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc8b4147060 con 0x7fc8b40aed80 2024-03-30T07:16:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-03-30T07:16:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 -- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f10341d99e0 msgr2=0x7f10341d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f10341d99e0 0x7f10341d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 -- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c003970 msgr2=0x7f1034141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c003970 0x7f1034141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 monclient: found mon.noname-b 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.026+0000 7f1031d9b640 1 -- 172.21.15.47:0/3963426815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10341da540 con 0x7f10341d6600 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.029+0000 7f1031d9b640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f102800c680 tx=0x7f102800cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.029+0000 7f1022ffd640 10 client.?.objecter ms_handle_connect 0x7f10341d6600 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.029+0000 7f1022ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:53.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1028005270 con 0x7f10341d6600 2024-03-30T07:16:53.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:53.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.3 2024-03-30T07:16:53.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:53.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 10 monclient: dump: 2024-03-30T07:16:53.039 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:53.039 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:53.039 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:53.040 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10280053d0 con 0x7f10341d6600 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1038e69640 5 monclient: authenticate success, global_id 4348 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1038e69640 10 monclient: _renew_subs 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1038e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.032+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10340aa7f0 con 0x7f10341d6600 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 monclient: _renew_subs 2024-03-30T07:16:53.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10340aad00 con 0x7f10341d6600 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 1 librados: init done 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1022ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102800a880 con 0x7f10341d6600 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1030d99640 4 set_mon_vals no callback set 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1022ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1022ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1022ffd640 10 monclient: dump: 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:53.042 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.033+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1018005040 con 0x7f10341d6600 2024-03-30T07:16:53.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.111+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f82a803b050 con 0x7f82b00b0120 2024-03-30T07:16:53.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.112+0000 7f8297fff640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 0x7f829c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:53.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.112+0000 7f82aeffd640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 0x7f829c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:53.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.112+0000 7f82aeffd640 10 monclient: get_auth_request con 0x7f829c06c5e0 auth_method 0 2024-03-30T07:16:53.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.112+0000 7f82aeffd640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 0x7f829c06eaa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005880 tx=0x7f82a0001150 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:53.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.112+0000 7f8297fff640 10 client.4317.objecter ms_handle_connect 0x7f829c06c5e0 2024-03-30T07:16:53.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f82a807b600 con 0x7f82b00b0120 2024-03-30T07:16:53.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 10 client.4317.objecter ms_dispatch 0x7f82b0000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 3 client.4317.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 3 client.4317.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.126+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f82a8077240 con 0x7f82b00b0120 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.127+0000 7f8297fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.127+0000 7f8297fff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:53.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.139+0000 7fc8b0ff9640 1 -- 172.21.15.47:0/1868753564 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8b4147060 con 0x7fc8b40aed80 2024-03-30T07:16:53.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.139+0000 7fc8b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:53.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.139+0000 7fc8b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.140 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-03-30T07:16:53.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 10 librados: watch_flush enter 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 10 librados: watch_flush exit 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 10 client.4320.objecter successfully canceled tick 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 msgr2=0x7fc8a006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 0x7fc8a006ea50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc8a40098f0 tx=0x7fc8a4008040 comp rx=0 tx=0).stop 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 10 monclient: shutdown 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 msgr2=0x7fc8b40a9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c430 tx=0x7fc8a800c8f0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 shutdown_connections 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc8a006c590 0x7fc8a006ea50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc8b4145bb0 0x7fc8b40a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc8b40aed80 0x7fc8b40a9400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 --2- 172.21.15.47:0/1868753564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc8b406d290 0x7fc8b40abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 >> 172.21.15.47:0/1868753564 conn(0x7fc8b40af610 msgr2=0x7fc8b4066270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 shutdown_connections 2024-03-30T07:16:53.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 -- 172.21.15.47:0/1868753564 wait complete. 2024-03-30T07:16:53.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.141+0000 7fc8b9570640 1 librados: shutdown 2024-03-30T07:16:53.154 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.5 2024-03-30T07:16:53.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.241+0000 7fac68819640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:53.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.241+0000 7fac68819640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:53.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.241+0000 7fac68819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.241+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fac5c030280 con 0x7fac5c0c5770 2024-03-30T07:16:53.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.301+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f102800a9e0 con 0x7f10341d6600 2024-03-30T07:16:53.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.301+0000 7f1022ffd640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 0x7f101406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:53.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.302+0000 7f103259c640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 0x7f101406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:53.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.302+0000 7f103259c640 10 monclient: get_auth_request con 0x7f101406c5e0 auth_method 0 2024-03-30T07:16:53.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.302+0000 7f103259c640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 0x7f101406eaa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1024036000 tx=0x7f1024030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:53.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.302+0000 7f1022ffd640 10 client.4348.objecter ms_handle_connect 0x7f101406c5e0 2024-03-30T07:16:53.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.305+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1028058670 con 0x7f10341d6600 2024-03-30T07:16:53.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.305+0000 7f1022ffd640 10 client.4348.objecter ms_dispatch 0x7f1034000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:53.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.305+0000 7f1022ffd640 3 client.4348.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:53.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.305+0000 7f1022ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:53.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.305+0000 7f1022ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-30T07:16:53.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.316+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f10280542b0 con 0x7f10341d6600 2024-03-30T07:16:53.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.317+0000 7f1022ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.317+0000 7f1022ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:53.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.325+0000 7fac4b7fe640 1 -- 172.21.15.47:0/3502651278 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac5c030280 con 0x7fac5c0c5770 2024-03-30T07:16:53.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.325+0000 7fac4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:53.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.325+0000 7fac4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.327 INFO:teuthology.orchestra.run.smithi047.stdout:60129542151 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.327+0000 7fac68819640 10 librados: watch_flush enter 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.327+0000 7fac68819640 10 librados: watch_flush exit 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.327+0000 7fac68819640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 10 client.4339.objecter successfully canceled tick 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 msgr2=0x7fac5006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 0x7fac5006ea80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac54009a10 tx=0x7fac54008040 comp rx=0 tx=0).stop 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 10 monclient: shutdown 2024-03-30T07:16:53.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 msgr2=0x7fac5c15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fac580090b0 tx=0x7fac580046c0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 shutdown_connections 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac5006c5c0 0x7fac5006ea80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac5c030560 0x7fac5c15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac5c0c5770 0x7fac5c15c8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 --2- 172.21.15.47:0/3502651278 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac5c038d80 0x7fac5c02fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 >> 172.21.15.47:0/3502651278 conn(0x7fac5c0065e0 msgr2=0x7fac5c0c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 shutdown_connections 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 -- 172.21.15.47:0/3502651278 wait complete. 2024-03-30T07:16:53.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.328+0000 7fac68819640 1 librados: shutdown 2024-03-30T07:16:53.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.6 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb8bc0582f0 con 0x7fb8c8140500 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 10 client.4268.objecter ms_dispatch 0x7fb8c8000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 3 client.4268.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 3 client.4268.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb8bc053f30 con 0x7fb8c8140500 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.348+0000 7fb8b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.356+0000 7f82b59b1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.356+0000 7f82b59b1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.356+0000 7f82b59b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.356+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f82b0144180 con 0x7f82b00b0120 2024-03-30T07:16:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.360+0000 7f8297fff640 1 -- 172.21.15.47:0/1865941726 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f82a804e090 con 0x7f82b00b0120 2024-03-30T07:16:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.360+0000 7f8297fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:53.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.360+0000 7f8297fff640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.361 INFO:teuthology.orchestra.run.smithi047.stdout:81604378631 2024-03-30T07:16:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 10 librados: watch_flush enter 2024-03-30T07:16:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 10 librados: watch_flush exit 2024-03-30T07:16:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 10 client.4317.objecter successfully canceled tick 2024-03-30T07:16:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 msgr2=0x7f829c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 0x7f829c06eaa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005880 tx=0x7f82a0001150 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 10 monclient: shutdown 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 msgr2=0x7f82b01da050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009d90 tx=0x7f82a8004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 shutdown_connections 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f829c06c5e0 0x7f829c06eaa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f82b00b0d20 0x7f82b013bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f82b00b0120 0x7f82b01da050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 --2- 172.21.15.47:0/1865941726 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f82b00a84a0 0x7f82b01d9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 >> 172.21.15.47:0/1865941726 conn(0x7f82b00b70b0 msgr2=0x7f82b00b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 shutdown_connections 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 -- 172.21.15.47:0/1865941726 wait complete. 2024-03-30T07:16:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.363+0000 7f82b59b1640 1 librados: shutdown 2024-03-30T07:16:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.368+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa908057420 con 0x7fa910033e70 2024-03-30T07:16:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.368+0000 7fa90e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.368+0000 7fa90e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.374+0000 7f3126ffd640 1 -- 172.21.15.47:0/414862862 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f312807aad0 con 0x7f3130144ab0 2024-03-30T07:16:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.374+0000 7f3126ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.374+0000 7f3126ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.375 INFO:teuthology.orchestra.run.smithi047.stdout:94489280521 2024-03-30T07:16:53.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 10 librados: watch_flush enter 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 10 librados: watch_flush exit 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 10 client.4250.objecter successfully canceled tick 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 msgr2=0x7f311406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 0x7f311406ea50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3118006fd0 tx=0x7f3118008040 comp rx=0 tx=0).stop 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 10 monclient: shutdown 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 msgr2=0x7f313013cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.376+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3128009f40 tx=0x7f3128004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 shutdown_connections 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f311406c590 0x7f311406ea50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3130144ab0 0x7f313013cdb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f31300b0d20 0x7f31300ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 --2- 172.21.15.47:0/414862862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f31300b0120 0x7f31300aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 >> 172.21.15.47:0/414862862 conn(0x7f31300b70b0 msgr2=0x7f3130068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 shutdown_connections 2024-03-30T07:16:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 -- 172.21.15.47:0/414862862 wait complete. 2024-03-30T07:16:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.377+0000 7f31349ab640 1 librados: shutdown 2024-03-30T07:16:53.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378631 for osd.7 2024-03-30T07:16:53.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280521 for osd.0 2024-03-30T07:16:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.538+0000 7f1038e69640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.538+0000 7f1038e69640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.538+0000 7f1038e69640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.538+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f10180055c0 con 0x7f10341d6600 2024-03-30T07:16:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7f1022ffd640 1 -- 172.21.15.47:0/3963426815 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1028017070 con 0x7f10341d6600 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7f1022ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7f1022ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stdout:85899345928 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7fb8ccd2a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7fb8ccd2a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7fb8ccd2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:53.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.542+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb8ac0055c0 con 0x7fb8c8140500 2024-03-30T07:16:53.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 10 librados: watch_flush enter 2024-03-30T07:16:53.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 10 librados: watch_flush exit 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 10 client.4348.objecter successfully canceled tick 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 msgr2=0x7f101406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 0x7f101406eaa0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1024036000 tx=0x7f1024030000 comp rx=0 tx=0).stop 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 10 monclient: shutdown 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 msgr2=0x7f10341d6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f102800c680 tx=0x7f102800cb40 comp rx=0 tx=0).stop 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 shutdown_connections 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f101406c5e0 0x7f101406eaa0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f10341d99e0 0x7f10341d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f10341d6600 0x7f10341d6a20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 --2- 172.21.15.47:0/3963426815 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f101c003970 0x7f1034141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 >> 172.21.15.47:0/3963426815 conn(0x7f10340b70b0 msgr2=0x7f10340b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 shutdown_connections 2024-03-30T07:16:53.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 -- 172.21.15.47:0/3963426815 wait complete. 2024-03-30T07:16:53.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.544+0000 7f1038e69640 1 librados: shutdown 2024-03-30T07:16:53.558 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345928 for osd.4 2024-03-30T07:16:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.584+0000 7fb8b6ffd640 1 -- 172.21.15.47:0/3415949614 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8bc017070 con 0x7fb8c8140500 2024-03-30T07:16:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.584+0000 7fb8b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.584+0000 7fb8b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.585 INFO:teuthology.orchestra.run.smithi047.stdout:94489280522 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 10 librados: watch_flush enter 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 10 librados: watch_flush exit 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 10 client.4268.objecter successfully canceled tick 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 msgr2=0x7fb8a806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 0x7fb8a806ead0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8034000 tx=0x7fb8b802e000 comp rx=0 tx=0).stop 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 10 monclient: shutdown 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 msgr2=0x7fb8c80ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c440 tx=0x7fb8bc00c900 comp rx=0 tx=0).stop 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 shutdown_connections 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb8a806c610 0x7fb8a806ead0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb8c81d6bf0 0x7fb8c81d7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb8c8140500 0x7fb8c80ac0f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 --2- 172.21.15.47:0/3415949614 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb8c8144ab0 0x7fb8c80ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.586+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 >> 172.21.15.47:0/3415949614 conn(0x7fb8c80b70b0 msgr2=0x7fb8c80b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.587+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 shutdown_connections 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.587+0000 7fb8ccd2a640 1 -- 172.21.15.47:0/3415949614 wait complete. 2024-03-30T07:16:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.587+0000 7fb8ccd2a640 1 librados: shutdown 2024-03-30T07:16:53.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280523 got 94489280522 for osd.2 2024-03-30T07:16:53.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.616+0000 7fa91ea22640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:53.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.616+0000 7fa91ea22640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:53.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.616+0000 7fa91ea22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:53.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.616+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa9100056d0 con 0x7fa910033e70 2024-03-30T07:16:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.620+0000 7fa90e7fc640 1 -- 172.21.15.47:0/657022570 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa908017070 con 0x7fa910033e70 2024-03-30T07:16:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.620+0000 7fa90e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:53.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.620+0000 7fa90e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:53.620 INFO:teuthology.orchestra.run.smithi047.stdout:68719476743 2024-03-30T07:16:53.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 10 librados: watch_flush enter 2024-03-30T07:16:53.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 10 librados: watch_flush exit 2024-03-30T07:16:53.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 10 client.4265.objecter successfully canceled tick 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 msgr2=0x7fa8fc06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 0x7fa8fc06ea50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa914009670 tx=0x7fa914006bf0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 10 monclient: shutdown 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 msgr2=0x7fa9100bbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c440 tx=0x7fa90800c900 comp rx=0 tx=0).stop 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 shutdown_connections 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa8fc06c590 0x7fa8fc06ea50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa910037570 0x7fa9100bc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa910033e70 0x7fa9100bbc60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 --2- 172.21.15.47:0/657022570 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa91002a8f0 0x7fa9100c10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 >> 172.21.15.47:0/657022570 conn(0x7fa91003a8e0 msgr2=0x7fa9100c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 shutdown_connections 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 -- 172.21.15.47:0/657022570 wait complete. 2024-03-30T07:16:53.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:53.622+0000 7fa91ea22640 1 librados: shutdown 2024-03-30T07:16:53.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.1 2024-03-30T07:16:54.038 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.141+0000 7f04c33a2640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.141+0000 7f04c33a2640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.141+0000 7f04c33a2640 10 monclient: monmap: 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.142195+0000 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.142195+0000 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.142 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.143 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.143 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.143 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.143 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.143 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 Processor -- start 2024-03-30T07:16:54.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 -- start start 2024-03-30T07:16:54.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient: init 2024-03-30T07:16:54.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04bc0a8920 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04bc0b36d0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc14ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-b con 0x7f04bc1450b0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0aa1a0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0aa410 con 0x7f04bc0a8920 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0b3cf0 con 0x7f04bc1450b0 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.142+0000 7f04c33a2640 10 monclient(hunting): authenticate will time out at 1540.257978s 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c1b9f640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f04bc0b36d0 0x7f04bc1449a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40830/0 (socket says 172.21.15.47:40830) 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c1b9f640 1 -- 172.21.15.47:0/224509581 learned_addr learned my addr 172.21.15.47:0/224509581 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:54.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c2ba1640 1 --2- 172.21.15.47:0/224509581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc14ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c2ba1640 10 monclient(hunting): get_auth_request con 0x7f04bc1450b0 auth_method 0 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c2ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c2ba1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.143+0000 7f04c2ba1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.144+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.144+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.144+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540381762 0 0) 0x7f04bc0aa410 con 0x7f04bc0a8920 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _negotiate 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): my global_id is 4351 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a8003520 con 0x7f04bc0a8920 2024-03-30T07:16:54.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310327666 0 0) 0x7f04bc0aa1a0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _negotiate 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 10 monclient(hunting): my global_id is 4326 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.145+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc0aa410 con 0x7f04bc0b36d0 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 1 -- 172.21.15.47:0/224509581 >> v1:172.21.15.52:6789/0 conn(0x7f04bc0a8920 legacy=0x7f04bc13e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 1 -- 172.21.15.47:0/224509581 >> v1:172.21.15.47:6789/0 conn(0x7f04bc0b36d0 legacy=0x7f04bc1449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.146+0000 7f04c2ba1640 1 -- 172.21.15.47:0/224509581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc145500 con 0x7f04bc1450b0 2024-03-30T07:16:54.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.148+0000 7f04c2ba1640 1 --2- 172.21.15.47:0/224509581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc14ccb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009dc0 tx=0x7f04b402be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537760fb3724176a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b402cbb0 con 0x7f04bc1450b0 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: dump: 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.153 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b402cd10 con 0x7f04bc1450b0 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c33a2640 5 monclient: authenticate success, global_id 4271 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 1 -- 172.21.15.47:0/224509581 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4034660 con 0x7f04bc1450b0 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c33a2640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c33a2640 4 set_mon_vals no callback set 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.152+0000 7f04c139e640 10 monclient: dump: 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.154 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.155 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.155 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T07:16:54.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 10 monclient: shutdown 2024-03-30T07:16:54.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 -- 172.21.15.47:0/224509581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 msgr2=0x7f04bc14ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 --2- 172.21.15.47:0/224509581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc14ccb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009dc0 tx=0x7f04b402be00 comp rx=0 tx=0).stop 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 -- 172.21.15.47:0/224509581 shutdown_connections 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 --2- 172.21.15.47:0/224509581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc14ccb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 -- 172.21.15.47:0/224509581 >> 172.21.15.47:0/224509581 conn(0x7f04bc0659e0 msgr2=0x7f04bc065df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 -- 172.21.15.47:0/224509581 shutdown_connections 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.153+0000 7f04c33a2640 1 -- 172.21.15.47:0/224509581 wait complete. 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 10 monclient: monmap: 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.160 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 librados: starting msgr at 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 librados: starting objecter 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 Processor -- start 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 -- start start 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 librados: setting wanted keys 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 1 librados: calling monclient init 2024-03-30T07:16:54.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.154+0000 7f04c33a2640 10 monclient: init 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04bc0a8920 0x7f04bc13a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-b con 0x7f04bc0a8920 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 0x7f04bc1d6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-a con 0x7f04bc0b36d0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc1d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): picked mon.noname-c con 0x7f04bc1450b0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc13f2e0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc138a30 con 0x7f04bc0a8920 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc13d340 con 0x7f04bc1450b0 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c33a2640 10 monclient(hunting): authenticate will time out at 1540.270322s 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc1d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c1b9f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 0x7f04bc1d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc1d6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54994/0 (socket says 172.21.15.47:54994) 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 1 -- 172.21.15.47:0/3145006395 learned_addr learned my addr 172.21.15.47:0/3145006395 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 10 monclient(hunting): get_auth_request con 0x7f04bc1450b0 auth_method 0 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c2ba1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c1b9f640 10 monclient(hunting): get_auth_request con 0x7f04bc0b36d0 auth_method 0 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c1b9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c1b9f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c1b9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.155+0000 7f04c23a0640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04bc0a8920 0x7f04bc13a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): get_auth_request con 0x7f04bc0a8920 auth_method 0 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c2ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c1b9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c1b9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c1b9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.156+0000 7f04c23a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-03-30T07:16:54.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 -- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 msgr2=0x7f04bc1d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc1d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 -- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04bc0a8920 msgr2=0x7f04bc13a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04bc0a8920 0x7f04bc13a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 monclient: found mon.noname-a 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.159+0000 7f04c1b9f640 1 -- 172.21.15.47:0/3145006395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009a70 con 0x7f04bc0b36d0 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.162+0000 7f04c1b9f640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 0x7f04bc1d6980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3e0 tx=0x7f04ac00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.162+0000 7f04b2ffd640 10 client.?.objecter ms_handle_connect 0x7f04bc0b36d0 2024-03-30T07:16:54.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.162+0000 7f04b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac005ce0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: dump: 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.169 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac005e40 con 0x7f04bc0b36d0 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac0072d0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04b2ffd640 10 monclient: dump: 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c0b9d640 4 set_mon_vals no callback set 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c33a2640 5 monclient: authenticate success, global_id 4329 2024-03-30T07:16:54.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c33a2640 10 monclient: _renew_subs 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c33a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc1d63b0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.165+0000 7f04c33a2640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.166+0000 7f04c33a2640 10 monclient: _renew_subs 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.166+0000 7f04c33a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.166+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc13e0d0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.166+0000 7f04c33a2640 1 librados: init done 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.167+0000 7f04c33a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.167+0000 7f04c33a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.167+0000 7f04c33a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.167+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc13f450 con 0x7f04bc0b36d0 2024-03-30T07:16:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.187+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f04ac0074f0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.187+0000 7f04b2ffd640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 0x7f04a006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.188+0000 7f04c23a0640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 0x7f04a006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.188+0000 7f04c23a0640 10 monclient: get_auth_request con 0x7f04a006c5e0 auth_method 0 2024-03-30T07:16:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.188+0000 7f04b2ffd640 10 client.4329.objecter ms_handle_connect 0x7f04a006c5e0 2024-03-30T07:16:54.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.188+0000 7f04c23a0640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 0x7f04a006eaa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04b80011e0 tx=0x7f04b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 10 monclient: monmap: 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.261856+0000 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.261856+0000 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.262 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.263 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.263 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.263 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 1 Processor -- start 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 1 -- start start 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.261+0000 7fe3451f3640 10 monclient: init 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3400b4280 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-b con 0x7fe34013e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe340141670 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34013f2b0 con 0x7fe3400b4280 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3400aa3c0 con 0x7fe340141670 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3400b4660 con 0x7fe34013e930 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.262+0000 7fe3451f3640 10 monclient(hunting): authenticate will time out at 1540.378105s 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:54996/0 (socket says 172.21.15.47:54996) 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 1 -- 172.21.15.47:0/3494690283 learned_addr learned my addr 172.21.15.47:0/3494690283 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe34013e930 auth_method 0 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.263+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.264+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.264+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.264+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.265+0000 7fe337fff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 1 -- 172.21.15.47:0/3494690283 >> v1:172.21.15.52:6789/0 conn(0x7fe340141670 legacy=0x7fe340141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 1 -- 172.21.15.47:0/3494690283 >> v1:172.21.15.47:6789/0 conn(0x7fe3400b4280 legacy=0x7fe3400a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.266+0000 7fe337fff640 1 -- 172.21.15.47:0/3494690283 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400aa630 con 0x7fe34013e930 2024-03-30T07:16:54.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.267+0000 7fe337fff640 1 --2- 172.21.15.47:0/3494690283 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3280098a0 tx=0x7fe32802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a30a295a541bb671 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 1 -- 172.21.15.47:0/3494690283 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328031640 con 0x7fe34013e930 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: dump: 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.273 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 1 -- 172.21.15.47:0/3494690283 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3280317a0 con 0x7fe34013e930 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 1 -- 172.21.15.47:0/3494690283 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328031b30 con 0x7fe34013e930 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe33e59c640 10 monclient: dump: 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.274 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe3451f3640 5 monclient: authenticate success, global_id 4277 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe3451f3640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.272+0000 7fe3451f3640 4 set_mon_vals no callback set 2024-03-30T07:16:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 10 monclient: shutdown 2024-03-30T07:16:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 -- 172.21.15.47:0/3494690283 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 msgr2=0x7fe34013ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3494690283 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe3280098a0 tx=0x7fe32802b8f0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 -- 172.21.15.47:0/3494690283 shutdown_connections 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3494690283 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe34013e930 0x7fe34013ed70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 -- 172.21.15.47:0/3494690283 >> 172.21.15.47:0/3494690283 conn(0x7fe340065e00 msgr2=0x7fe340066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 -- 172.21.15.47:0/3494690283 shutdown_connections 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.273+0000 7fe3451f3640 1 -- 172.21.15.47:0/3494690283 wait complete. 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.274+0000 7fe3451f3640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.274+0000 7fe3451f3640 10 monclient: monmap: 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.279 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 librados: starting msgr at 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 librados: starting objecter 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 Processor -- start 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 -- start start 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 librados: setting wanted keys 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 1 librados: calling monclient init 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.275+0000 7fe3451f3640 10 monclient: init 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3400b4280 0x7fe340140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-b con 0x7fe3400b4280 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34013e930 0x7fe34012a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-a con 0x7fe34013e930 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): picked mon.noname-c con 0x7fe340141670 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340144560 con 0x7fe34013e930 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340133580 con 0x7fe3400b4280 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3401339a0 con 0x7fe340141670 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe3451f3640 10 monclient(hunting): authenticate will time out at 1540.391463s 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.276+0000 7fe33f59e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:55012/0 (socket says 172.21.15.47:55012) 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 1 -- 172.21.15.47:0/3655046445 learned_addr learned my addr 172.21.15.47:0/3655046445 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): get_auth_request con 0x7fe340141670 auth_method 0 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3400b4280 0x7fe340140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): get_auth_request con 0x7fe3400b4280 auth_method 0 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.277+0000 7fe33ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-03-30T07:16:54.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 -- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3400b4280 msgr2=0x7fe340140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3400b4280 0x7fe340140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 -- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34013e930 msgr2=0x7fe34012a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34013e930 0x7fe34012a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 monclient: found mon.noname-c 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.280+0000 7fe33f59e640 1 -- 172.21.15.47:0/3655046445 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280094d0 con 0x7fe340141670 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.282+0000 7fe33f59e640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c6b0 tx=0x7fe33800cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.282+0000 7fe3377fe640 10 client.?.objecter ms_handle_connect 0x7fe340141670 2024-03-30T07:16:54.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.282+0000 7fe3377fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338003b70 con 0x7fe340141670 2024-03-30T07:16:54.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:54.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: dump: 2024-03-30T07:16:54.287 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 5 monclient: authenticate success, global_id 4280 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe338003cf0 con 0x7fe340141670 2024-03-30T07:16:54.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe338003ea0 con 0x7fe340141670 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3377fe640 10 monclient: dump: 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe33dd9b640 4 set_mon_vals no callback set 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 10 monclient: _renew_subs 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe340142c50 con 0x7fe340141670 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 10 monclient: _renew_subs 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe340143820 con 0x7fe340141670 2024-03-30T07:16:54.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.286+0000 7fe3451f3640 1 librados: init done 2024-03-30T07:16:54.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.288+0000 7fe3451f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.288+0000 7fe3451f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.288+0000 7fe3451f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.288+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3401443c0 con 0x7fe340141670 2024-03-30T07:16:54.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.306+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe338005310 con 0x7fe340141670 2024-03-30T07:16:54.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.306+0000 7fe3377fe640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 0x7fe32006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.306+0000 7fe33ed9d640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 0x7fe32006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.306+0000 7fe33ed9d640 10 monclient: get_auth_request con 0x7fe32006c5e0 auth_method 0 2024-03-30T07:16:54.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.307+0000 7fe33ed9d640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 0x7fe32006eaa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3300017f0 tx=0x7fe330009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.307+0000 7fe3377fe640 10 client.4280.objecter ms_handle_connect 0x7fe32006c5e0 2024-03-30T07:16:54.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.310+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe338058f10 con 0x7fe340141670 2024-03-30T07:16:54.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.310+0000 7fe3377fe640 10 client.4280.objecter ms_dispatch 0x7fe340000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:54.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.310+0000 7fe3377fe640 3 client.4280.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:54.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.310+0000 7fe3377fe640 3 client.4280.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:54.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.310+0000 7fe3377fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-30T07:16:54.341 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T07:16:54.381 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T07:16:54.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T07:16:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.449+0000 7f265c04b640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.449+0000 7f265c04b640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.449+0000 7f265c04b640 10 monclient: monmap: 2024-03-30T07:16:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.450003+0000 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.450003+0000 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.450+0000 7f265c04b640 1 Processor -- start 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.450+0000 7f265c04b640 1 -- start start 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.450+0000 7f265c04b640 10 monclient: init 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26540b0120 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 0x7f2654144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-b con 0x7f26540a84a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26540b0d20 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540a8900 con 0x7f26540b0d20 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540a8bd0 con 0x7f26540b0120 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26540aa4f0 con 0x7f26540a84a0 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.451+0000 7f265c04b640 10 monclient(hunting): authenticate will time out at 1540.566975s 2024-03-30T07:16:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.452+0000 7f2659de7640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f26540b0120 0x7f26540712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:56310/0 (socket says 172.21.15.47:56310) 2024-03-30T07:16:54.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.452+0000 7f2659de7640 1 -- 172.21.15.47:0/4228449349 learned_addr learned my addr 172.21.15.47:0/4228449349 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:54.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285921445 0 0) 0x7f26540a8bd0 con 0x7f26540b0120 2024-03-30T07:16:54.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 10 monclient(hunting): _negotiate 2024-03-30T07:16:54.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 10 monclient(hunting): my global_id is 4363 2024-03-30T07:16:54.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.455+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2640003520 con 0x7f26540b0120 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f04ac057e90 con 0x7f04bc0b36d0 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 10 client.4329.objecter ms_dispatch 0x7f04bc000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 3 client.4329.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 3 client.4329.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f04ac0541d0 con 0x7f04bc0b36d0 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f04b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136877768 0 0) 0x7f2640003520 con 0x7f26540b0120 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 msgr2=0x7f2654144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 0x7f2654144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 >> v1:172.21.15.47:6789/0 conn(0x7f26540b0d20 legacy=0x7f265414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 monclient: found mon.noname-c-legacy 2024-03-30T07:16:54.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.52:6789/0 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540b3780 con 0x7f26540b0120 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f2650001670 con 0x7f26540b0120 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: dump: 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.462 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] but i'm connected to v1:172.21.15.52:6789/0, reconnecting 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 >> v1:172.21.15.52:6789/0 conn(0x7f26540b0120 legacy=0x7f26540712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 0x7f2640003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient(hunting): picked mon.c con 0x7f26540a84a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004060 0x7f26400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 10 monclient(hunting): picked mon.a con 0x7f2640004060 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.459+0000 7f2658de5640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640004d50 0x7f26400051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): picked mon.b con 0x7f2640004d50 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540aa4f0 con 0x7f2640004060 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264000bbc0 con 0x7f2640004d50 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264000bd80 con 0x7f26540a84a0 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.460+0000 7f2658de5640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.461+0000 7f265a5e8640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640004d50 0x7f26400051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): get_auth_request con 0x7f2640004d50 auth_method 0 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f265a5e8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.462+0000 7f26595e6640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004060 0x7f26400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.463+0000 7f26595e6640 10 monclient(hunting): get_auth_request con 0x7f2640004060 auth_method 0 2024-03-30T07:16:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.463+0000 7f26595e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.463+0000 7f26595e6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.463+0000 7f26595e6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 10 monclient(hunting): handle_auth_done global_id 4363 payload 293 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 -- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 msgr2=0x7f2640003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540a84a0 0x7f2640003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 -- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004060 msgr2=0x7f26400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004060 0x7f26400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 monclient: found mon.b 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 1 -- 172.21.15.47:0/4228449349 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540b3780 con 0x7f2640004d50 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265a5e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265c04b640 5 monclient: authenticate success, global_id 4363 2024-03-30T07:16:54.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.464+0000 7f265c04b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.467+0000 7f265a5e8640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640004d50 0x7f26400051b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f264c001d50 tx=0x7f26400057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7f7a9a7699415e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c003990 con 0x7f2640004d50 2024-03-30T07:16:54.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: dump: 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.474 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c003b10 con 0x7f2640004d50 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 1 -- 172.21.15.47:0/4228449349 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c005070 con 0x7f2640004d50 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f2658de5640 10 monclient: dump: 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.475 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f265c04b640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.471+0000 7f265c04b640 4 set_mon_vals no callback set 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 10 monclient: shutdown 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 -- 172.21.15.47:0/4228449349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640004d50 msgr2=0x7f26400051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 --2- 172.21.15.47:0/4228449349 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f2640004d50 0x7f26400051b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f264c001d50 tx=0x7f26400057f0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265a5e8640 1 -- 172.21.15.47:0/4228449349 reap_dead start 2024-03-30T07:16:54.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 -- 172.21.15.47:0/4228449349 shutdown_connections 2024-03-30T07:16:54.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 -- 172.21.15.47:0/4228449349 >> 172.21.15.47:0/4228449349 conn(0x7f26540b70b0 msgr2=0x7f26540b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 -- 172.21.15.47:0/4228449349 shutdown_connections 2024-03-30T07:16:54.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.472+0000 7f265c04b640 1 -- 172.21.15.47:0/4228449349 wait complete. 2024-03-30T07:16:54.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.473+0000 7f265c04b640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.473+0000 7f265c04b640 10 monclient: monmap: 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.473+0000 7f265c04b640 1 librados: starting msgr at 2024-03-30T07:16:54.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 1 librados: starting objecter 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 1 Processor -- start 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 1 -- start start 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 1 librados: setting wanted keys 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 1 librados: calling monclient init 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.474+0000 7f265c04b640 10 monclient: init 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004d50 0x7f26540aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-a con 0x7f2640004d50 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-b con 0x7f26540abb50 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540ac720 0x7f26541d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): picked mon.noname-c con 0x7f26540ac720 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540ab710 con 0x7f2640004d50 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540ab2f0 con 0x7f26540abb50 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26541d9d20 con 0x7f26540ac720 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f265c04b640 10 monclient(hunting): authenticate will time out at 1540.590467s 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.475+0000 7f26595e6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f2659de7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004d50 0x7f26540aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44290/0 (socket says 172.21.15.47:44290) 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 1 -- 172.21.15.47:0/864241197 learned_addr learned my addr 172.21.15.47:0/864241197 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): get_auth_request con 0x7f26540abb50 auth_method 0 2024-03-30T07:16:54.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f2659de7640 10 monclient(hunting): get_auth_request con 0x7f2640004d50 auth_method 0 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f2659de7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f2659de7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f2659de7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.476+0000 7f26595e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.477+0000 7f2659de7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.477+0000 7f2659de7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.477+0000 7f2659de7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 -- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540ac720 msgr2=0x7f26541d97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540ac720 0x7f26541d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 -- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004d50 msgr2=0x7f26540aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004d50 0x7f26540aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.479+0000 7f26595e6640 1 -- 172.21.15.47:0/864241197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26541d9ee0 con 0x7f26540abb50 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.482+0000 7f26595e6640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f264400e7c0 tx=0x7f264400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.482+0000 7f264a7fc640 10 client.?.objecter ms_handle_connect 0x7f26540abb50 2024-03-30T07:16:54.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.482+0000 7f264a7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644004030 con 0x7f26540abb50 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: dump: 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26440041b0 con 0x7f26540abb50 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f265c04b640 5 monclient: authenticate success, global_id 4366 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400c730 con 0x7f26540abb50 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264a7fc640 10 monclient: dump: 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.488 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.489 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.489 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.489 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.489 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.486+0000 7f264bfff640 4 set_mon_vals no callback set 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _renew_subs 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26541da130 con 0x7f26540abb50 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _renew_subs 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265413cf50 con 0x7f26540abb50 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 1 librados: init done 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.487+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26540ab850 con 0x7f26540abb50 2024-03-30T07:16:54.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 10 monclient: monmap: 2024-03-30T07:16:54.495 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.495877+0000 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.495877+0000 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.496 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 1 Processor -- start 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 1 -- start start 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.495+0000 7f964f199640 10 monclient: init 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96480b4280 0x7f96480a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): picked mon.noname-b con 0x7f96480b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f964813e930 0x7f964813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): picked mon.noname-a con 0x7f964813e930 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9648141670 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964813f2b0 con 0x7f9648141670 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96480b4660 con 0x7f964813e930 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480aa3c0 con 0x7f96480b4280 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.496+0000 7f964f199640 10 monclient(hunting): authenticate will time out at 1540.611966s 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f964813e930 0x7f964813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f964813e930 0x7f964813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51170/0 (socket says 172.21.15.47:51170) 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 1 -- 172.21.15.47:0/1731898588 learned_addr learned my addr 172.21.15.47:0/1731898588 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 10 monclient(hunting): get_auth_request con 0x7f964813e930 auth_method 0 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.497+0000 7f9647fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.498+0000 7f9647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.498+0000 7f9647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.498+0000 7f9647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 1 -- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96480b4280 msgr2=0x7f96480a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 1 --2- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96480b4280 0x7f96480a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 1 -- 172.21.15.47:0/1731898588 >> v1:172.21.15.52:6789/0 conn(0x7f9648141670 legacy=0x7f9648141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 1 monclient: found mon.noname-a 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.499+0000 7f9647fff640 1 -- 172.21.15.47:0/1731898588 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480aa630 con 0x7f964813e930 2024-03-30T07:16:54.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.502+0000 7f9647fff640 1 --2- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f964813e930 0x7f964813ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9634009870 tx=0x7f963402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c5ceee7c148f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.505+0000 7f96477fe640 1 -- 172.21.15.47:0/1731898588 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96340317c0 con 0x7f964813e930 2024-03-30T07:16:54.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.505+0000 7f96477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.505+0000 7f96477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.505+0000 7f96477fe640 10 monclient: dump: 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 1 -- 172.21.15.47:0/1731898588 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9634031920 con 0x7f964813e930 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 5 monclient: authenticate success, global_id 4335 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 4 set_mon_vals no callback set 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 1 -- 172.21.15.47:0/1731898588 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634031cb0 con 0x7f964813e930 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f96477fe640 10 monclient: dump: 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.509 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 10 monclient: shutdown 2024-03-30T07:16:54.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 -- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f964813e930 msgr2=0x7f964813ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 --2- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f964813e930 0x7f964813ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9634009870 tx=0x7f963402b860 comp rx=0 tx=0).stop 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 -- 172.21.15.47:0/1731898588 shutdown_connections 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 --2- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f964813e930 0x7f964813ed70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 --2- 172.21.15.47:0/1731898588 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96480b4280 0x7f96480a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 -- 172.21.15.47:0/1731898588 >> 172.21.15.47:0/1731898588 conn(0x7f9648065e00 msgr2=0x7f9648066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 -- 172.21.15.47:0/1731898588 shutdown_connections 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.506+0000 7f964f199640 1 -- 172.21.15.47:0/1731898588 wait complete. 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.507+0000 7f964f199640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.507+0000 7f964f199640 10 monclient: monmap: 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.512 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 librados: starting msgr at 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 librados: starting objecter 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 Processor -- start 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 -- start start 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 librados: setting wanted keys 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 1 librados: calling monclient init 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.508+0000 7f964f199640 10 monclient: init 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:54.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96480b4280 0x7f9648136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): picked mon.noname-b con 0x7f96480b4280 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f964813e930 0x7f9648135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): picked mon.noname-c con 0x7f964813e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): picked mon.noname-a con 0x7f9648141670 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648136580 con 0x7f9648141670 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648138240 con 0x7f96480b4280 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964813ddb0 con 0x7f964813e930 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964f199640 10 monclient(hunting): authenticate will time out at 1540.624500s 2024-03-30T07:16:54.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.509+0000 7f964d736640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51186/0 (socket says 172.21.15.47:51186) 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 1 -- 172.21.15.47:0/704475871 learned_addr learned my addr 172.21.15.47:0/704475871 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): get_auth_request con 0x7f9648141670 auth_method 0 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.510+0000 7f964d736640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 -- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f964813e930 msgr2=0x7f9648135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f964813e930 0x7f9648135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 -- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96480b4280 msgr2=0x7f9648136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96480b4280 0x7f9648136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 monclient: found mon.noname-a 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.513+0000 7f964d736640 1 -- 172.21.15.47:0/704475871 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340094d0 con 0x7f9648141670 2024-03-30T07:16:54.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.516+0000 7f964d736640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f964000b9c0 tx=0x7f964000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.516+0000 7f96457fa640 10 client.?.objecter ms_handle_connect 0x7f9648141670 2024-03-30T07:16:54.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.516+0000 7f96457fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964000cba0 con 0x7f9648141670 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 10 monclient: dump: 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.519+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964000cd00 con 0x7f9648141670 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f96457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640014680 con 0x7f9648141670 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f96457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f96457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f96457fa640 10 monclient: dump: 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 5 monclient: authenticate success, global_id 4338 2024-03-30T07:16:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 10 monclient: _renew_subs 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f9646ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9648136200 con 0x7f9648141670 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 10 monclient: _renew_subs 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96481d6d00 con 0x7f9648141670 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.520+0000 7f964f199640 1 librados: init done 2024-03-30T07:16:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.521+0000 7f964f199640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.521+0000 7f964f199640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.521+0000 7f964f199640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.521+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648136390 con 0x7f9648141670 2024-03-30T07:16:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.542+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f964001d3b0 con 0x7f9648141670 2024-03-30T07:16:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.542+0000 7f96457fa640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 0x7f962c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7f964cf35640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 0x7f962c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7f964cf35640 10 monclient: get_auth_request con 0x7f962c06c5e0 auth_method 0 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7f96457fa640 10 client.4338.objecter ms_handle_connect 0x7f962c06c5e0 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7f964cf35640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 0x7f962c06eaa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f963c009710 tx=0x7f963c006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7fb46cfb1640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7fb46cfb1640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.543+0000 7fb46cfb1640 10 monclient: monmap: 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.544487+0000 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.544487+0000 2024-03-30T07:16:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.545 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.544+0000 7fb46cfb1640 1 Processor -- start 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.544+0000 7fb46cfb1640 1 -- start start 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.544+0000 7fb46cfb1640 10 monclient: init 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4680b4280 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb46813e930 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb46813eda0 0x7fb468141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-c con 0x7fb46813eda0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680aa3c0 con 0x7fb46813e930 2024-03-30T07:16:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680aa630 con 0x7fb4680b4280 2024-03-30T07:16:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4680b4660 con 0x7fb46813eda0 2024-03-30T07:16:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.545+0000 7fb46cfb1640 10 monclient(hunting): authenticate will time out at 1540.660357s 2024-03-30T07:16:54.559 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T07:16:54.602 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T07:16:54.635 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T07:16:54.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.663+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fe338054b50 con 0x7fe340141670 2024-03-30T07:16:54.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.663+0000 7fe3377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.663+0000 7fe3377fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:54.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.675+0000 7f20dd284640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.675+0000 7f20dd284640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.675+0000 7f20dd284640 10 monclient: monmap: 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.676291+0000 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.676291+0000 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.677 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.676+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9640058d90 con 0x7f9648141670 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.676+0000 7f96457fa640 10 client.4338.objecter ms_dispatch 0x7f9648000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.676+0000 7f96457fa640 3 client.4338.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.676+0000 7f96457fa640 3 client.4338.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:54.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.676+0000 7f96457fa640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.677+0000 7f20dd284640 1 Processor -- start 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.677+0000 7f20dd284640 1 -- start start 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.677+0000 7f20dd284640 10 monclient: init 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f20d80ae360 0x7f20d806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-c con 0x7f20d80ae360 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20d80ae740 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f20d8070d10 0x7f20d8071120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-a con 0x7f20d8070d10 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d8071660 con 0x7f20d80ae740 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d80702a0 con 0x7f20d8070d10 2024-03-30T07:16:54.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d80708c0 con 0x7f20d80ae360 2024-03-30T07:16:54.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.678+0000 7f20dd284640 10 monclient(hunting): authenticate will time out at 1540.793347s 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.732+0000 7f791b456640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.732+0000 7f791b456640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.732+0000 7f791b456640 10 monclient: monmap: 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.733623+0000 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.733623+0000 2024-03-30T07:16:54.734 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.733+0000 7f791b456640 1 Processor -- start 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.733+0000 7f791b456640 1 -- start start 2024-03-30T07:16:54.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.733+0000 7f791b456640 10 monclient: init 2024-03-30T07:16:54.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79140b0120 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f79140a84a0 0x7f7914144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): picked mon.noname-c con 0x7f79140a84a0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f79140b0d20 0x7f791414be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): picked mon.noname-a con 0x7f79140b0d20 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79140a8900 con 0x7f79140b0120 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79140aa4f0 con 0x7f79140b0d20 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79140a8bd0 con 0x7f79140a84a0 2024-03-30T07:16:54.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.734+0000 7f791b456640 10 monclient(hunting): authenticate will time out at 1540.849819s 2024-03-30T07:16:54.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.757+0000 7f328a81a640 10 monclient: get_monmap_and_config 2024-03-30T07:16:54.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.757+0000 7f328a81a640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.757+0000 7f328a81a640 10 monclient: monmap: 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:54.758334+0000 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:54.758334+0000 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.759 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:54.760 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 1 Processor -- start 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 1 -- start start 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 10 monclient: init 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f32840b4280 0x7f32840a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-c con 0x7f32840b4280 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f328413e930 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.758+0000 7f328a81a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-b con 0x7f328413eda0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32840aa3c0 con 0x7f328413e930 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840b4660 con 0x7f32840b4280 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840aa630 con 0x7f328413eda0 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f328a81a640 10 monclient(hunting): authenticate will time out at 1540.874167s 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.759+0000 7f3288db7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:55060/0 (socket says 172.21.15.47:55060) 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 1 -- 172.21.15.47:0/526104757 learned_addr learned my addr 172.21.15.47:0/526104757 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 10 monclient(hunting): get_auth_request con 0x7f328413eda0 auth_method 0 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.760+0000 7f3288db7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.761+0000 7f3288db7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.761+0000 7f3288db7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.761+0000 7f3288db7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.762+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f264400c890 con 0x7f26540abb50 2024-03-30T07:16:54.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.762+0000 7f264a7fc640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 0x7f263806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.762+0000 7f2659de7640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 0x7f263806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.762+0000 7f2659de7640 10 monclient: get_auth_request con 0x7f263806c590 auth_method 0 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f264a7fc640 10 client.4366.objecter ms_handle_connect 0x7f263806c590 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f2659de7640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 0x7f263806ea50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2650004d10 tx=0x7f2650001b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 1 -- 172.21.15.47:0/526104757 >> v2:172.21.15.52:3300/0 conn(0x7f32840b4280 msgr2=0x7f32840a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 1 --2- 172.21.15.47:0/526104757 >> v2:172.21.15.52:3300/0 conn(0x7f32840b4280 0x7f32840a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 1 -- 172.21.15.47:0/526104757 >> v1:172.21.15.47:6789/0 conn(0x7f328413e930 legacy=0x7f3284141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.763+0000 7f3288db7640 1 -- 172.21.15.47:0/526104757 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840a9210 con 0x7f328413eda0 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.765+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2644058850 con 0x7f26540abb50 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.765+0000 7f264a7fc640 10 client.4366.objecter ms_dispatch 0x7f2654000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:54.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.765+0000 7f264a7fc640 3 client.4366.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:54.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.765+0000 7f264a7fc640 3 client.4366.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:54.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.765+0000 7f264a7fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-30T07:16:54.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.766+0000 7fb466d9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb46813eda0 0x7fb468141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.766+0000 7f3288db7640 1 --2- 172.21.15.47:0/526104757 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f327c009e50 tx=0x7f327c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb17efb094be4e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.770+0000 7f3282ffd640 1 -- 172.21.15.47:0/526104757 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c02c9e0 con 0x7f328413eda0 2024-03-30T07:16:54.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.770+0000 7f3282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.770+0000 7f3282ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.770+0000 7f3282ffd640 10 monclient: dump: 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 1 -- 172.21.15.47:0/526104757 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c02cb40 con 0x7f328413eda0 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 5 monclient: authenticate success, global_id 4283 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 1 -- 172.21.15.47:0/526104757 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c0343f0 con 0x7f328413eda0 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 4 set_mon_vals no callback set 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f3282ffd640 10 monclient: dump: 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.773 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.774 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 10 monclient: shutdown 2024-03-30T07:16:54.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 -- 172.21.15.47:0/526104757 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 msgr2=0x7f3284141d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 --2- 172.21.15.47:0/526104757 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f327c009e50 tx=0x7f327c02be40 comp rx=0 tx=0).stop 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 -- 172.21.15.47:0/526104757 shutdown_connections 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 --2- 172.21.15.47:0/526104757 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f328413eda0 0x7f3284141d80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 --2- 172.21.15.47:0/526104757 >> v2:172.21.15.52:3300/0 conn(0x7f32840b4280 0x7f32840a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 -- 172.21.15.47:0/526104757 >> 172.21.15.47:0/526104757 conn(0x7f3284065e00 msgr2=0x7f3284066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 -- 172.21.15.47:0/526104757 shutdown_connections 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.771+0000 7f328a81a640 1 -- 172.21.15.47:0/526104757 wait complete. 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.772+0000 7f328a81a640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.772+0000 7f328a81a640 10 monclient: monmap: 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.777 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.778 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 librados: starting msgr at 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 librados: starting objecter 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 Processor -- start 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 -- start start 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 librados: setting wanted keys 2024-03-30T07:16:54.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 librados: calling monclient init 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 10 monclient: init 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-c con 0x7f32840b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f328413e930 0x7f3284139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-b con 0x7f328413e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.773+0000 7f328a81a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f328413eda0 0x7f3284136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): picked mon.noname-a con 0x7f328413eda0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284141290 con 0x7f328413eda0 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32841d6610 con 0x7f328413e930 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3284137030 con 0x7f32840b4280 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f328a81a640 10 monclient(hunting): authenticate will time out at 1540.889207s 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:55070/0 (socket says 172.21.15.47:55070) 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 1 -- 172.21.15.47:0/3957490581 learned_addr learned my addr 172.21.15.47:0/3957490581 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 10 monclient(hunting): get_auth_request con 0x7f32840b4280 auth_method 0 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.774+0000 7f3283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.775+0000 7f3283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.775+0000 7f3283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.775+0000 7f3283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb46813eda0 0x7fb468141d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44300/0 (socket says 172.21.15.47:44300) 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 1 -- 172.21.15.47:0/423096532 learned_addr learned my addr 172.21.15.47:0/423096532 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 10 monclient(hunting): get_auth_request con 0x7fb46813eda0 auth_method 0 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7fb466d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 -- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f328413e930 msgr2=0x7f3284139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f328413e930 0x7f3284139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 -- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f328413eda0 msgr2=0x7f3284136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f328413eda0 0x7f3284136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 monclient: found mon.noname-c 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f3283fff640 1 -- 172.21.15.47:0/3957490581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c009b30 con 0x7f32840b4280 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f2644054b90 con 0x7f26540abb50 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f264a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.778+0000 7f264a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7f20d6ffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f20d80ae360 0x7f20d806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7f3283fff640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f327400b4a0 tx=0x7f327400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7f3280ff9640 10 client.?.objecter ms_handle_connect 0x7f32840b4280 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.780+0000 7f3280ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f20d80ae360 0x7f20d806ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44310/0 (socket says 172.21.15.47:44310) 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 1 -- 172.21.15.47:0/2862923177 learned_addr learned my addr 172.21.15.47:0/2862923177 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 10 monclient(hunting): get_auth_request con 0x7f20d80ae360 auth_method 0 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.781+0000 7f20d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 1 -- 172.21.15.47:0/423096532 >> v1:172.21.15.52:6789/0 conn(0x7fb4680b4280 legacy=0x7fb4680a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 1 -- 172.21.15.47:0/423096532 >> v1:172.21.15.47:6789/0 conn(0x7fb46813e930 legacy=0x7fb468141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 1 monclient: found mon.noname-c 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.782+0000 7fb466d9d640 1 -- 172.21.15.47:0/423096532 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4680a9210 con 0x7fb46813eda0 2024-03-30T07:16:54.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.784+0000 7f20d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.784+0000 7f20d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.784+0000 7f20d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.785+0000 7fb466d9d640 1 --2- 172.21.15.47:0/423096532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb468141d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb4680b6ff0 tx=0x7fb46002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=643b9d77fad19bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-03-30T07:16:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 1 -- 172.21.15.47:0/2862923177 >> v2:172.21.15.47:3300/0 conn(0x7f20d8070d10 msgr2=0x7f20d8071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 1 --2- 172.21.15.47:0/2862923177 >> v2:172.21.15.47:3300/0 conn(0x7f20d8070d10 0x7f20d8071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 1 -- 172.21.15.47:0/2862923177 >> v1:172.21.15.47:6789/0 conn(0x7f20d80ae740 legacy=0x7f20d813e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 1 monclient: found mon.noname-c 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.786+0000 7f20d6ffd640 1 -- 172.21.15.47:0/2862923177 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d813dc10 con 0x7f20d80ae360 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.789+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274011830 con 0x7f32840b4280 2024-03-30T07:16:54.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: dump: 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 5 monclient: authenticate success, global_id 4286 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 10 monclient: _renew_subs 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 1 -- 172.21.15.47:0/3957490581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32841d6b30 con 0x7f32840b4280 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 10 monclient: _renew_subs 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 1 -- 172.21.15.47:0/3957490581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32841d7090 con 0x7f32840b4280 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f328a81a640 1 librados: init done 2024-03-30T07:16:54.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f20d6ffd640 1 --2- 172.21.15.47:0/2862923177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80ae360 0x7f20d806ed40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20d80720b0 tx=0x7f20c802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7ef33bb52a154af server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274011990 con 0x7f32840b4280 2024-03-30T07:16:54.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f3280ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.790+0000 7f32827fc640 4 set_mon_vals no callback set 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f326a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274011830 con 0x7f32840b4280 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f3280ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f3280ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f3280ff9640 10 monclient: dump: 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.800 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f326a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f326a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.791+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c005040 con 0x7f32840b4280 2024-03-30T07:16:54.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.792+0000 7f79189f1640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f79140a84a0 0x7f7914144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f79140a84a0 0x7f7914144ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44318/0 (socket says 172.21.15.47:44318) 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 1 -- 172.21.15.47:0/1542480989 learned_addr learned my addr 172.21.15.47:0/1542480989 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 10 monclient(hunting): get_auth_request con 0x7f79140a84a0 auth_method 0 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.793+0000 7f79189f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.795+0000 7f79189f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.795+0000 7f79189f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.795+0000 7f79189f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 1 -- 172.21.15.47:0/423096532 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46002c9e0 con 0x7fb46813eda0 2024-03-30T07:16:54.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:54.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: dump: 2024-03-30T07:16:54.804 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.804 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 1 -- 172.21.15.47:0/423096532 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46002cb40 con 0x7fb46813eda0 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 1 -- 172.21.15.47:0/423096532 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4600343f0 con 0x7fb46813eda0 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46559a640 10 monclient: dump: 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.806 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46cfb1640 5 monclient: authenticate success, global_id 4369 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46cfb1640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.796+0000 7fb46cfb1640 4 set_mon_vals no callback set 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 10 monclient: shutdown 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 -- 172.21.15.47:0/423096532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 msgr2=0x7fb468141d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/423096532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb468141d80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb4680b6ff0 tx=0x7fb46002be40 comp rx=0 tx=0).stop 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 -- 172.21.15.47:0/423096532 shutdown_connections 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/423096532 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb468141d80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 -- 172.21.15.47:0/423096532 >> 172.21.15.47:0/423096532 conn(0x7fb468065e00 msgr2=0x7fb468066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 -- 172.21.15.47:0/423096532 shutdown_connections 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.797+0000 7fb46cfb1640 1 -- 172.21.15.47:0/423096532 wait complete. 2024-03-30T07:16:54.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.798+0000 7fb46cfb1640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.798+0000 7fb46cfb1640 10 monclient: monmap: 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 librados: starting msgr at 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 librados: starting objecter 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 Processor -- start 2024-03-30T07:16:54.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 -- start start 2024-03-30T07:16:54.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 librados: setting wanted keys 2024-03-30T07:16:54.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 1 librados: calling monclient init 2024-03-30T07:16:54.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7fb46cfb1640 10 monclient: init 2024-03-30T07:16:54.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 1 -- 172.21.15.47:0/1542480989 >> v2:172.21.15.47:3300/0 conn(0x7f79140b0d20 msgr2=0x7f791414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 1 --2- 172.21.15.47:0/1542480989 >> v2:172.21.15.47:3300/0 conn(0x7f79140b0d20 0x7f791414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 1 -- 172.21.15.47:0/1542480989 >> v1:172.21.15.52:6789/0 conn(0x7f79140b0120 legacy=0x7f79140712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 1 monclient: found mon.noname-c 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.799+0000 7f79189f1640 1 -- 172.21.15.47:0/1542480989 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79140b3780 con 0x7f79140a84a0 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4680b4280 0x7fb468139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-c con 0x7fb4680b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46813e930 0x7fb4680aad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-a con 0x7fb46813e930 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): picked mon.noname-b con 0x7fb46813eda0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680ac1e0 con 0x7fb46813e930 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680ac420 con 0x7fb46813eda0 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46813a1b0 con 0x7fb4680b4280 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.800+0000 7fb46cfb1640 10 monclient(hunting): authenticate will time out at 1540.915733s 2024-03-30T07:16:54.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.804+0000 7f79189f1640 1 --2- 172.21.15.47:0/1542480989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140a84a0 0x7f7914144ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7914071ea0 tx=0x7f790402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=533d640179ed08ea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 1 -- 172.21.15.47:0/2862923177 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8031750 con 0x7f20d80ae360 2024-03-30T07:16:54.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: dump: 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20dd284640 5 monclient: authenticate success, global_id 4372 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20dd284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 1 -- 172.21.15.47:0/2862923177 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c80318b0 con 0x7f20d80ae360 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20dd284640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.805+0000 7f20dd284640 4 set_mon_vals no callback set 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 10 monclient: shutdown 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 -- 172.21.15.47:0/2862923177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80ae360 msgr2=0x7f20d806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20d5ffb640 1 -- 172.21.15.47:0/2862923177 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8031750 con 0x7f20d80ae360 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 --2- 172.21.15.47:0/2862923177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80ae360 0x7f20d806ed40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20d80720b0 tx=0x7f20c802b7f0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20d5ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 -- 172.21.15.47:0/2862923177 shutdown_connections 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 --2- 172.21.15.47:0/2862923177 >> v2:172.21.15.47:3300/0 conn(0x7f20d8070d10 0x7f20d8071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 --2- 172.21.15.47:0/2862923177 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80ae360 0x7f20d806ed40 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f20d80720b0 tx=0x7f20c802b7f0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 -- 172.21.15.47:0/2862923177 >> 172.21.15.47:0/2862923177 conn(0x7f20d80b30d0 msgr2=0x7f20d80b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 -- 172.21.15.47:0/2862923177 shutdown_connections 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.806+0000 7f20dd284640 1 -- 172.21.15.47:0/2862923177 wait complete. 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.807+0000 7f20dd284640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.807+0000 7f20dd284640 10 monclient: monmap: 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.818 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.807+0000 7f20dd284640 1 librados: starting msgr at 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 1 librados: starting objecter 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 1 Processor -- start 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 1 -- start start 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 1 librados: setting wanted keys 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 1 librados: calling monclient init 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.808+0000 7f20dd284640 10 monclient: init 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20d8070d10 0x7f20d80a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-c con 0x7f20d8070d10 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-b con 0x7f20d80a9be0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20d80acef0 0x7f20d81e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): picked mon.noname-a con 0x7f20d80acef0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d80ab2e0 con 0x7f20d80acef0 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d80aa1a0 con 0x7f20d80a9be0 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d80b2250 con 0x7f20d8070d10 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.809+0000 7f20dd284640 10 monclient(hunting): authenticate will time out at 1540.924776s 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.810+0000 7fb466d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44334/0 (socket says 172.21.15.47:44334) 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 1 -- 172.21.15.47:0/3920033756 learned_addr learned my addr 172.21.15.47:0/3920033756 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 10 monclient(hunting): get_auth_request con 0x7fb46813eda0 auth_method 0 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7fb466d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44342/0 (socket says 172.21.15.47:44342) 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 1 -- 172.21.15.47:0/2191850486 learned_addr learned my addr 172.21.15.47:0/2191850486 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 10 monclient(hunting): get_auth_request con 0x7f20d80a9be0 auth_method 0 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.811+0000 7f20d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.812+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.812+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.812+0000 7fb466d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 1 -- 172.21.15.47:0/1542480989 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904031870 con 0x7f79140a84a0 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: dump: 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.823 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 1 -- 172.21.15.47:0/1542480989 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79040319d0 con 0x7f79140a84a0 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 1 -- 172.21.15.47:0/1542480989 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904031d60 con 0x7f79140a84a0 2024-03-30T07:16:54.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f790bfff640 10 monclient: dump: 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f791b456640 5 monclient: authenticate success, global_id 4375 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f791b456640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f791b456640 4 set_mon_vals no callback set 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f20d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f20d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.813+0000 7f20d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 10 monclient: shutdown 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 -- 172.21.15.47:0/1542480989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140a84a0 msgr2=0x7f7914144ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 --2- 172.21.15.47:0/1542480989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140a84a0 0x7f7914144ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7914071ea0 tx=0x7f790402b910 comp rx=0 tx=0).stop 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 -- 172.21.15.47:0/1542480989 shutdown_connections 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 --2- 172.21.15.47:0/1542480989 >> v2:172.21.15.47:3300/0 conn(0x7f79140b0d20 0x7f791414be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 --2- 172.21.15.47:0/1542480989 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140a84a0 0x7f7914144ab0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 -- 172.21.15.47:0/1542480989 >> 172.21.15.47:0/1542480989 conn(0x7f79140b70b0 msgr2=0x7f79140b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 -- 172.21.15.47:0/1542480989 shutdown_connections 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.814+0000 7f791b456640 1 -- 172.21.15.47:0/1542480989 wait complete. 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.815+0000 7f791b456640 10 monclient: build_initial_monmap 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.815+0000 7f791b456640 10 monclient: monmap: 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.829 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 librados: starting msgr at 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 librados: starting objecter 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 Processor -- start 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 -- start start 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 librados: setting wanted keys 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 1 librados: calling monclient init 2024-03-30T07:16:54.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7f791b456640 10 monclient: init 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 -- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4680b4280 msgr2=0x7fb468139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4680b4280 0x7fb468139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 -- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46813e930 msgr2=0x7fb4680aad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46813e930 0x7fb4680aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.816+0000 7fb466d9d640 1 -- 172.21.15.47:0/3920033756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460009b30 con 0x7fb46813eda0 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.817+0000 7f791b456640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.817+0000 7f791b456640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 0x7f79140b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): picked mon.noname-a con 0x7f79140a84a0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f79140b0120 0x7f7914141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): picked mon.noname-c con 0x7f79140b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 0x7f79141d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): picked mon.noname-b con 0x7f79140b0d20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:54.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914141310 con 0x7f79140a84a0 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914141ad0 con 0x7f79140b0d20 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.818+0000 7f791b456640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.819+0000 7f791b456640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7914141d70 con 0x7f79140b0120 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.819+0000 7f791b456640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.819+0000 7f791b456640 10 monclient(hunting): authenticate will time out at 1540.934293s 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-03-30T07:16:54.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 -- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20d8070d10 msgr2=0x7f20d80a8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20d8070d10 0x7f20d80a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 -- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20d80acef0 msgr2=0x7f20d81e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20d80acef0 0x7f20d81e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 monclient: found mon.noname-b 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.820+0000 7f20d67fc640 1 -- 172.21.15.47:0/2191850486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c80094d0 con 0x7f20d80a9be0 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.821+0000 7fb466d9d640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb460036040 tx=0x7fb460004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.821+0000 7fb456ffd640 10 client.?.objecter ms_handle_connect 0x7fb46813eda0 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.821+0000 7fb456ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.827+0000 7f20d67fc640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00c410 tx=0x7f20cc00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7f20bf7fe640 10 client.?.objecter ms_handle_connect 0x7f20d80a9be0 2024-03-30T07:16:54.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7f20bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46003e450 con 0x7fb46813eda0 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: dump: 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.837 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46003e5b0 con 0x7fb46813eda0 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46003e9a0 con 0x7fb46813eda0 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb456ffd640 10 monclient: dump: 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb464d99640 4 set_mon_vals no callback set 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 5 monclient: authenticate success, global_id 4378 2024-03-30T07:16:54.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 10 monclient: _renew_subs 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46813b160 con 0x7fb46813eda0 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 10 monclient: _renew_subs 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46813ad80 con 0x7fb46813eda0 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.829+0000 7fb46cfb1640 1 librados: init done 2024-03-30T07:16:54.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.828+0000 7f79199f3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 0x7f79141d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7fb46cfb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7fb46cfb1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7fb46cfb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46813af70 con 0x7fb46813eda0 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 0x7f79141d64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44358/0 (socket says 172.21.15.47:44358) 2024-03-30T07:16:54.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 1 -- 172.21.15.47:0/3731367400 learned_addr learned my addr 172.21.15.47:0/3731367400 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 10 monclient(hunting): get_auth_request con 0x7f79140b0d20 auth_method 0 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.830+0000 7f79199f3640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.831+0000 7f04c33a2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.831+0000 7f04c33a2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.831+0000 7f04c33a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.831+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f04bc13f420 con 0x7f04bc0b36d0 2024-03-30T07:16:54.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.852+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb46003eb00 con 0x7fb46813eda0 2024-03-30T07:16:54.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.853+0000 7fb456ffd640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 0x7fb44806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:54.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.853+0000 7fb46659c640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 0x7fb44806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.853+0000 7fb46659c640 10 monclient: get_auth_request con 0x7fb44806c590 auth_method 0 2024-03-30T07:16:54.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.853+0000 7fb46659c640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 0x7fb44806ea50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb458001560 tx=0x7fb458007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.853+0000 7fb456ffd640 10 client.4378.objecter ms_handle_connect 0x7fb44806c590 2024-03-30T07:16:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.892+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f96400549d0 con 0x7f9648141670 2024-03-30T07:16:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.892+0000 7f96457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.892+0000 7f96457fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.893+0000 7f79191f2640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 0x7f79140b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.895+0000 7f79191f2640 10 monclient(hunting): get_auth_request con 0x7f79140a84a0 auth_method 0 2024-03-30T07:16:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.895+0000 7f79191f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.895+0000 7f79191f2640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.895+0000 7f79191f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.897+0000 7f79191f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.897+0000 7f79191f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.897+0000 7f79191f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-03-30T07:16:54.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 -- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f79140b0120 msgr2=0x7f7914141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f79140b0120 0x7f7914141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 -- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 msgr2=0x7f79141d64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 0x7f79141d64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 10 monclient: _finish_hunting 0 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 monclient: found mon.noname-a 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.901+0000 7f79191f2640 1 -- 172.21.15.47:0/3731367400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040094d0 con 0x7f79140a84a0 2024-03-30T07:16:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.905+0000 7f79191f2640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 0x7f79140b6320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f791000c410 tx=0x7f791000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.905+0000 7f7909ffb640 10 client.?.objecter ms_handle_connect 0x7f79140a84a0 2024-03-30T07:16:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.905+0000 7f7909ffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:54.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79100035f0 con 0x7f79140a84a0 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: dump: 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.919 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7910003770 con 0x7f79140a84a0 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910003b20 con 0x7f79140a84a0 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f7909ffb640 10 monclient: dump: 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:54.920 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f790b7fe640 4 set_mon_vals no callback set 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f791b456640 5 monclient: authenticate success, global_id 4341 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f791b456640 10 monclient: _renew_subs 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f791b456640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.915+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79141d4000 con 0x7f79140a84a0 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.916+0000 7f791b456640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.916+0000 7f791b456640 10 monclient: _renew_subs 2024-03-30T07:16:54.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.916+0000 7f791b456640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.916+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7914142470 con 0x7f79140a84a0 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.916+0000 7f791b456640 1 librados: init done 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.917+0000 7f791b456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.917+0000 7f791b456640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.917+0000 7f791b456640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:54.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.917+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791404e030 con 0x7f79140a84a0 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7fe3451f3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7fe3451f3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7fe3451f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe3401da120 con 0x7fe340141670 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7f04b2ffd640 1 -- 172.21.15.47:0/3145006395 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f04ac021020 con 0x7f04bc0b36d0 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7f04b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:54.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.919+0000 7f04b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:54.924 INFO:teuthology.orchestra.run.smithi047.stdout:60129542148 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 10 librados: watch_flush enter 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 10 librados: watch_flush exit 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 10 client.4329.objecter successfully canceled tick 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 msgr2=0x7f04a006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 0x7f04a006eaa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f04b80011e0 tx=0x7f04b8009040 comp rx=0 tx=0).stop 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 10 monclient: shutdown 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 msgr2=0x7f04bc1d6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 0x7f04bc1d6980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04ac00c3e0 tx=0x7f04ac00c8a0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 shutdown_connections 2024-03-30T07:16:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f04a006c5e0 0x7f04a006eaa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04bc1450b0 0x7f04bc1d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04bc0b36d0 0x7f04bc1d6980 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 --2- 172.21.15.47:0/3145006395 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f04bc0a8920 0x7f04bc13a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 >> 172.21.15.47:0/3145006395 conn(0x7f04bc0659e0 msgr2=0x7f04bc0b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.922+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 shutdown_connections 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.923+0000 7f04c33a2640 1 -- 172.21.15.47:0/3145006395 wait complete. 2024-03-30T07:16:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:54.923+0000 7f04c33a2640 1 librados: shutdown 2024-03-30T07:16:54.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.3 2024-03-30T07:16:55.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.006+0000 7f265c04b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:55.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.006+0000 7f265c04b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:55.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.006+0000 7f265c04b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.006+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f265404e030 con 0x7f26540abb50 2024-03-30T07:16:55.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc0035f0 con 0x7f20d80a9be0 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: dump: 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:55.046 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20cc003750 con 0x7f20d80a9be0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 5 monclient: authenticate success, global_id 4381 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 10 monclient: _renew_subs 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d80a93a0 con 0x7f20d80a9be0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 10 monclient: _renew_subs 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d80ad510 con 0x7f20d80a9be0 2024-03-30T07:16:55.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20dd284640 1 librados: init done 2024-03-30T07:16:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.044+0000 7f20d57fa640 4 set_mon_vals no callback set 2024-03-30T07:16:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.046+0000 7f20dd284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.046+0000 7f20dd284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.046+0000 7f20dd284640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.046+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d804e030 con 0x7f20d80a9be0 2024-03-30T07:16:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.103+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc003b00 con 0x7f20d80a9be0 2024-03-30T07:16:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.103+0000 7f20bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.103+0000 7f20bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.103+0000 7f20bf7fe640 10 monclient: dump: 2024-03-30T07:16:55.104 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:55.105 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:55.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.107+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb46007bc70 con 0x7fb46813eda0 2024-03-30T07:16:55.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.107+0000 7fb456ffd640 10 client.4378.objecter ms_dispatch 0x7fb468000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:55.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.107+0000 7fb456ffd640 3 client.4378.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:55.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.107+0000 7fb456ffd640 3 client.4378.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:55.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.107+0000 7fb456ffd640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-30T07:16:55.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.118+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fb4600778b0 con 0x7fb46813eda0 2024-03-30T07:16:55.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.118+0000 7fb456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.118+0000 7fb456ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:55.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.124+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3274011b00 con 0x7f32840b4280 2024-03-30T07:16:55.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.125+0000 7f3280ff9640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 0x7f327006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:55.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.125+0000 7f32837fe640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 0x7f327006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:55.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.125+0000 7f32837fe640 10 monclient: get_auth_request con 0x7f327006c5f0 auth_method 0 2024-03-30T07:16:55.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.126+0000 7f3280ff9640 10 client.4286.objecter ms_handle_connect 0x7f327006c5f0 2024-03-30T07:16:55.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.126+0000 7f32837fe640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 0x7f327006eab0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32841390d0 tx=0x7f3278007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:55.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.127+0000 7f264a7fc640 1 -- 172.21.15.47:0/864241197 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2644020020 con 0x7f26540abb50 2024-03-30T07:16:55.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.127+0000 7f264a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:55.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.127+0000 7f264a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.128 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 10 librados: watch_flush enter 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 10 librados: watch_flush exit 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 10 client.4366.objecter successfully canceled tick 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 msgr2=0x7f263806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 0x7f263806ea50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2650004d10 tx=0x7f2650001b50 comp rx=0 tx=0).stop 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 10 monclient: shutdown 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 msgr2=0x7f26540ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f264400e7c0 tx=0x7f264400ec80 comp rx=0 tx=0).stop 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 shutdown_connections 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f263806c590 0x7f263806ea50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26540ac720 0x7f26541d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f26540abb50 0x7f26540ac1e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 --2- 172.21.15.47:0/864241197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2640004d50 0x7f26540aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 >> 172.21.15.47:0/864241197 conn(0x7f26540b70b0 msgr2=0x7f2654143a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 shutdown_connections 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 -- 172.21.15.47:0/864241197 wait complete. 2024-03-30T07:16:55.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.130+0000 7f265c04b640 1 librados: shutdown 2024-03-30T07:16:55.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.144+0000 7f964f199640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:55.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.144+0000 7f964f199640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:55.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.144+0000 7f964f199640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:55.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.144+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f964804e030 con 0x7f9648141670 2024-03-30T07:16:55.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.6 2024-03-30T07:16:55.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3274018310 con 0x7f32840b4280 2024-03-30T07:16:55.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 10 client.4286.objecter ms_dispatch 0x7f3284000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 3 client.4286.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 3 client.4286.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.150+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f3274054ab0 con 0x7f32840b4280 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.151+0000 7f3280ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.151+0000 7f3280ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:55.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.170+0000 7fe3377fe640 1 -- 172.21.15.47:0/3655046445 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe3401da120 con 0x7fe340141670 2024-03-30T07:16:55.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.170+0000 7fe3377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:55.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.170+0000 7fe3377fe640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.171 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.171+0000 7fe3451f3640 10 librados: watch_flush enter 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 10 librados: watch_flush exit 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 10 client.4280.objecter successfully canceled tick 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 msgr2=0x7fe32006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 0x7fe32006eaa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3300017f0 tx=0x7fe330009040 comp rx=0 tx=0).stop 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 10 monclient: shutdown 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 msgr2=0x7fe34012a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c6b0 tx=0x7fe33800cb70 comp rx=0 tx=0).stop 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 shutdown_connections 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fe32006c5e0 0x7fe32006eaa0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe340141670 0x7fe34012a540 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe34013e930 0x7fe34012a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 --2- 172.21.15.47:0/3655046445 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fe3400b4280 0x7fe340140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 >> 172.21.15.47:0/3655046445 conn(0x7fe340065e00 msgr2=0x7fe3400b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 shutdown_connections 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 -- 172.21.15.47:0/3655046445 wait complete. 2024-03-30T07:16:55.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.172+0000 7fe3451f3640 1 librados: shutdown 2024-03-30T07:16:55.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.5 2024-03-30T07:16:55.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.276+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7910003c80 con 0x7f79140a84a0 2024-03-30T07:16:55.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.276+0000 7f7909ffb640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 0x7f78f806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:55.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.277+0000 7f79189f1640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 0x7f78f806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:55.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.277+0000 7f79189f1640 10 monclient: get_auth_request con 0x7f78f806c590 auth_method 0 2024-03-30T07:16:55.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.277+0000 7f7909ffb640 10 client.4341.objecter ms_handle_connect 0x7f78f806c590 2024-03-30T07:16:55.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.277+0000 7f79189f1640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 0x7f78f806ea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7904036000 tx=0x7f7904030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:55.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7910057e50 con 0x7f79140a84a0 2024-03-30T07:16:55.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 10 client.4341.objecter ms_dispatch 0x7f7914000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:55.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 3 client.4341.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 3 client.4341.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-30T07:16:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.290+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7910053a90 con 0x7f79140a84a0 2024-03-30T07:16:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.291+0000 7f7909ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.291+0000 7f7909ffb640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.297+0000 7f96457fa640 1 -- 172.21.15.47:0/704475871 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9640019020 con 0x7f9648141670 2024-03-30T07:16:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.297+0000 7f96457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.297+0000 7f96457fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.298 INFO:teuthology.orchestra.run.smithi047.stdout:81604378632 2024-03-30T07:16:55.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 10 librados: watch_flush enter 2024-03-30T07:16:55.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 10 librados: watch_flush exit 2024-03-30T07:16:55.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 10 client.4338.objecter successfully canceled tick 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 msgr2=0x7f962c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 0x7f962c06eaa0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f963c009710 tx=0x7f963c006c90 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 10 monclient: shutdown 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 msgr2=0x7f96481d6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f964000b9c0 tx=0x7f964000be80 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 shutdown_connections 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f962c06c5e0 0x7f962c06eaa0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9648141670 0x7f96481d6580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f964813e930 0x7f9648135c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 --2- 172.21.15.47:0/704475871 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96480b4280 0x7f9648136a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 >> 172.21.15.47:0/704475871 conn(0x7f9648065e00 msgr2=0x7f9648133920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.299+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 shutdown_connections 2024-03-30T07:16:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.300+0000 7f964f199640 1 -- 172.21.15.47:0/704475871 wait complete. 2024-03-30T07:16:55.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.300+0000 7f964f199640 1 librados: shutdown 2024-03-30T07:16:55.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378632 for osd.7 2024-03-30T07:16:55.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.323+0000 7fb46cfb1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:55.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.323+0000 7fb46cfb1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:55.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.323+0000 7fb46cfb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.323+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb46813af70 con 0x7fb46813eda0 2024-03-30T07:16:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.348+0000 7f326a7fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.348+0000 7f326a7fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.348+0000 7f326a7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:55.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.348+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f325c0055c0 con 0x7f32840b4280 2024-03-30T07:16:55.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.352+0000 7f3280ff9640 1 -- 172.21.15.47:0/3957490581 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f32740580d0 con 0x7f32840b4280 2024-03-30T07:16:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.352+0000 7f3280ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:55.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.352+0000 7f3280ff9640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.353 INFO:teuthology.orchestra.run.smithi047.stdout:68719476743 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 10 librados: watch_flush enter 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 10 librados: watch_flush exit 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 10 client.4286.objecter successfully canceled tick 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 msgr2=0x7f327006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 0x7f327006eab0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32841390d0 tx=0x7f3278007990 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 10 monclient: shutdown 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 msgr2=0x7f3284135d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f327400b4a0 tx=0x7f327400b960 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 shutdown_connections 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f327006c5f0 0x7f327006eab0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f328413eda0 0x7f3284136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f328413e930 0x7f3284139b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 --2- 172.21.15.47:0/3957490581 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f32840b4280 0x7f3284135d50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 >> 172.21.15.47:0/3957490581 conn(0x7f3284065e00 msgr2=0x7f32840b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 shutdown_connections 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 -- 172.21.15.47:0/3957490581 wait complete. 2024-03-30T07:16:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.354+0000 7f326a7fc640 1 librados: shutdown 2024-03-30T07:16:55.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.1 2024-03-30T07:16:55.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.395+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f20cc003c60 con 0x7f20d80a9be0 2024-03-30T07:16:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.396+0000 7f20bf7fe640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 0x7f20c406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.396+0000 7f20d6ffd640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 0x7f20c406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.396+0000 7f20d6ffd640 10 monclient: get_auth_request con 0x7f20c406c5e0 auth_method 0 2024-03-30T07:16:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.396+0000 7f20d6ffd640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 0x7f20c406eaa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f20c8035000 tx=0x7f20c8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.396+0000 7f20bf7fe640 10 client.4381.objecter ms_handle_connect 0x7f20c406c5e0 2024-03-30T07:16:55.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.399+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f20cc0010c0 con 0x7f20d80a9be0 2024-03-30T07:16:55.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.399+0000 7f20bf7fe640 10 client.4381.objecter ms_dispatch 0x7f20d8000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:55.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.399+0000 7f20bf7fe640 3 client.4381.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:55.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.399+0000 7f20bf7fe640 3 client.4381.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:55.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.399+0000 7f20bf7fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-30T07:16:55.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.409+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f20cc053ef0 con 0x7f20d80a9be0 2024-03-30T07:16:55.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.409+0000 7f20bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:55.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.409+0000 7f20bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:55.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.463+0000 7fb456ffd640 1 -- 172.21.15.47:0/3920033756 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb460043090 con 0x7fb46813eda0 2024-03-30T07:16:55.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.463+0000 7fb456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-30T07:16:55.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.463+0000 7fb456ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.464 INFO:teuthology.orchestra.run.smithi047.stdout:94489280522 2024-03-30T07:16:55.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 10 librados: watch_flush enter 2024-03-30T07:16:55.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 10 librados: watch_flush exit 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 10 client.4378.objecter successfully canceled tick 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 msgr2=0x7fb44806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 0x7fb44806ea50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb458001560 tx=0x7fb458007990 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 10 monclient: shutdown 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 msgr2=0x7fb4680ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb460036040 tx=0x7fb460004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 shutdown_connections 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb44806c590 0x7fb44806ea50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb46813eda0 0x7fb4680ac840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb46813e930 0x7fb4680aad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 --2- 172.21.15.47:0/3920033756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4680b4280 0x7fb468139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 >> 172.21.15.47:0/3920033756 conn(0x7fb468065e00 msgr2=0x7fb46813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 shutdown_connections 2024-03-30T07:16:55.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 -- 172.21.15.47:0/3920033756 wait complete. 2024-03-30T07:16:55.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.466+0000 7fb46cfb1640 1 librados: shutdown 2024-03-30T07:16:55.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280522 got 94489280522 for osd.0 2024-03-30T07:16:55.482 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:55.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.488+0000 7f791b456640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:55.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.488+0000 7f791b456640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:55.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.488+0000 7f791b456640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:55.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.488+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7914142690 con 0x7f79140a84a0 2024-03-30T07:16:55.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.492+0000 7f7909ffb640 1 -- 172.21.15.47:0/3731367400 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7910017070 con 0x7f79140a84a0 2024-03-30T07:16:55.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.492+0000 7f7909ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-30T07:16:55.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.492+0000 7f7909ffb640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.493 INFO:teuthology.orchestra.run.smithi047.stdout:94489280523 2024-03-30T07:16:55.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 10 librados: watch_flush enter 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 10 librados: watch_flush exit 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 10 client.4341.objecter successfully canceled tick 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 msgr2=0x7f78f806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 0x7f78f806ea50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7904036000 tx=0x7f7904030000 comp rx=0 tx=0).stop 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 10 monclient: shutdown 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 msgr2=0x7f79140b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 0x7f79140b6320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f791000c410 tx=0x7f791000c810 comp rx=0 tx=0).stop 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 shutdown_connections 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f78f806c590 0x7f78f806ea50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f79140b0d20 0x7f79141d64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f79140b0120 0x7f7914141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 --2- 172.21.15.47:0/3731367400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f79140a84a0 0x7f79140b6320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 >> 172.21.15.47:0/3731367400 conn(0x7f79140b70b0 msgr2=0x7f79140b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 shutdown_connections 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 -- 172.21.15.47:0/3731367400 wait complete. 2024-03-30T07:16:55.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.494+0000 7f791b456640 1 librados: shutdown 2024-03-30T07:16:55.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280523 got 94489280523 for osd.2 2024-03-30T07:16:55.506 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:55.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.623+0000 7f20dd284640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:55.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.623+0000 7f20dd284640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:55.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.623+0000 7f20dd284640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:55.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.623+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f20d80ad510 con 0x7f20d80a9be0 2024-03-30T07:16:55.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.626+0000 7f20bf7fe640 1 -- 172.21.15.47:0/2191850486 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f20d80ad510 con 0x7f20d80a9be0 2024-03-30T07:16:55.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.626+0000 7f20bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-30T07:16:55.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.626+0000 7f20bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:55.627 INFO:teuthology.orchestra.run.smithi047.stdout:85899345930 2024-03-30T07:16:55.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 10 librados: watch_flush enter 2024-03-30T07:16:55.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 10 librados: watch_flush exit 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 10 client.4381.objecter successfully canceled tick 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 msgr2=0x7f20c406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 0x7f20c406eaa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f20c8035000 tx=0x7f20c8030000 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 10 monclient: shutdown 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 msgr2=0x7f20d80ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00c410 tx=0x7f20cc00c8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 shutdown_connections 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f20c406c5e0 0x7f20c406eaa0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f20d80acef0 0x7f20d81e3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f20d80a9be0 0x7f20d80ac9b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 --2- 172.21.15.47:0/2191850486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f20d8070d10 0x7f20d80a8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:55.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 >> 172.21.15.47:0/2191850486 conn(0x7f20d80b30d0 msgr2=0x7f20d8068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 shutdown_connections 2024-03-30T07:16:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 -- 172.21.15.47:0/2191850486 wait complete. 2024-03-30T07:16:55.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:55.629+0000 7f20dd284640 1 librados: shutdown 2024-03-30T07:16:55.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345930 for osd.4 2024-03-30T07:16:55.641 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:55.936 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T07:16:56.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.038+0000 7fc7c4c96640 10 monclient: get_monmap_and_config 2024-03-30T07:16:56.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.038+0000 7fc7c4c96640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.038+0000 7fc7c4c96640 10 monclient: monmap: 2024-03-30T07:16:56.039 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.039 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:56.039540+0000 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:56.039540+0000 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:56.040 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.039+0000 7fc7c4c96640 1 Processor -- start 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.039+0000 7fc7c4c96640 1 -- start start 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.039+0000 7fc7c4c96640 10 monclient: init 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7c00b4280 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fc7c013e930 0x7fc7c013eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-c con 0x7fc7c013e930 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7c0141670 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c013f2e0 con 0x7fc7c00b4280 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c00aa3c0 con 0x7fc7c0141670 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c00b4660 con 0x7fc7c013e930 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7c4c96640 10 monclient(hunting): authenticate will time out at 1542.155308s 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7be59c640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc7c00b4280 0x7fc7c00a8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40892/0 (socket says 172.21.15.47:40892) 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7be59c640 1 -- 172.21.15.47:0/3815222488 learned_addr learned my addr 172.21.15.47:0/3815222488 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.040+0000 7fc7bdd9b640 1 --2- 172.21.15.47:0/3815222488 >> v2:172.21.15.52:3300/0 conn(0x7fc7c013e930 0x7fc7c013eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.041+0000 7fc7bdd9b640 10 monclient(hunting): get_auth_request con 0x7fc7c013e930 auth_method 0 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.041+0000 7fc7bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.041+0000 7fc7bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.041+0000 7fc7bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 1 -- 172.21.15.47:0/3815222488 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487070044 0 0) 0x7fc7c013f2e0 con 0x7fc7c00b4280 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 10 monclient(hunting): _negotiate 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 10 monclient(hunting): my global_id is 4344 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.042+0000 7fc7bd59a640 1 -- 172.21.15.47:0/3815222488 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a8003520 con 0x7fc7c00b4280 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 1 -- 172.21.15.47:0/3815222488 >> v1:172.21.15.52:6789/0 conn(0x7fc7c0141670 legacy=0x7fc7c0141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 1 -- 172.21.15.47:0/3815222488 >> v1:172.21.15.47:6789/0 conn(0x7fc7c00b4280 legacy=0x7fc7c00a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 1 monclient: found mon.noname-c 2024-03-30T07:16:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.043+0000 7fc7bdd9b640 1 -- 172.21.15.47:0/3815222488 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00aa630 con 0x7fc7c013e930 2024-03-30T07:16:56.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.049+0000 7fc7bdd9b640 1 --2- 172.21.15.47:0/3815222488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c013e930 0x7fc7c013eda0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009d90 tx=0x7fc7b402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbd581a30d11b692 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 1 -- 172.21.15.47:0/3815222488 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b402cc60 con 0x7fc7c013e930 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: dump: 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.054 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 1 -- 172.21.15.47:0/3815222488 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b402cdc0 con 0x7fc7c013e930 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 1 -- 172.21.15.47:0/3815222488 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4034660 con 0x7fc7c013e930 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.052+0000 7fc7bd59a640 10 monclient: dump: 2024-03-30T07:16:56.055 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 5 monclient: authenticate success, global_id 4387 2024-03-30T07:16:56.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:56.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 4 set_mon_vals no callback set 2024-03-30T07:16:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 10 monclient: shutdown 2024-03-30T07:16:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 -- 172.21.15.47:0/3815222488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c013e930 msgr2=0x7fc7c013eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/3815222488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c013e930 0x7fc7c013eda0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009d90 tx=0x7fc7b402beb0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 -- 172.21.15.47:0/3815222488 shutdown_connections 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/3815222488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c013e930 0x7fc7c013eda0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 -- 172.21.15.47:0/3815222488 >> 172.21.15.47:0/3815222488 conn(0x7fc7c0065e00 msgr2=0x7fc7c0066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 -- 172.21.15.47:0/3815222488 shutdown_connections 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.053+0000 7fc7c4c96640 1 -- 172.21.15.47:0/3815222488 wait complete. 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 10 monclient: monmap: 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:56.061 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 librados: starting msgr at 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 librados: starting objecter 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 Processor -- start 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 -- start start 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 librados: setting wanted keys 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 1 librados: calling monclient init 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.054+0000 7fc7c4c96640 10 monclient: init 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc7c00b4280 0x7fc7c01456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-c con 0x7fc7c00b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 0x7fc7c0144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-a con 0x7fc7c013e930 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): picked mon.noname-b con 0x7fc7c0141670 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c0134f40 con 0x7fc7c013e930 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c0135190 con 0x7fc7c0141670 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c0135600 con 0x7fc7c00b4280 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7c4c96640 10 monclient(hunting): authenticate will time out at 1542.170575s 2024-03-30T07:16:56.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.055+0000 7fc7bdd9b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 0x7fc7c0144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7be59c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc7c00b4280 0x7fc7c01456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44384/0 (socket says 172.21.15.47:44384) 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 1 -- 172.21.15.47:0/576717488 learned_addr learned my addr 172.21.15.47:0/576717488 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bdd9b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 0x7fc7c0144e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51268/0 (socket says 172.21.15.47:51268) 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): get_auth_request con 0x7fc7c0141670 auth_method 0 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bdd9b640 10 monclient(hunting): get_auth_request con 0x7fc7c013e930 auth_method 0 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7be59c640 10 monclient(hunting): get_auth_request con 0x7fc7c00b4280 auth_method 0 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7be59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.056+0000 7fc7bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.057+0000 7fc7be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-03-30T07:16:56.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 -- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc7c00b4280 msgr2=0x7fc7c01456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc7c00b4280 0x7fc7c01456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 -- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 msgr2=0x7fc7c0144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 0x7fc7c0144e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 monclient: found mon.noname-b 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.059+0000 7fc7bed9d640 1 -- 172.21.15.47:0/576717488 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b4009a70 con 0x7fc7c0141670 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.062+0000 7fc7bed9d640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000c390 tx=0x7fc7b000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.062+0000 7fc7aeffd640 10 client.?.objecter ms_handle_connect 0x7fc7c0141670 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.062+0000 7fc7aeffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00052d0 con 0x7fc7c0141670 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: dump: 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.068 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b0005430 con 0x7fc7c0141670 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0003810 con 0x7fc7c0141670 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7aeffd640 10 monclient: dump: 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.071 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 5 monclient: authenticate success, global_id 4393 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 10 monclient: _renew_subs 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c013e270 con 0x7fc7c0141670 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7bcd99640 4 set_mon_vals no callback set 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:56.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 10 monclient: _renew_subs 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c01d7710 con 0x7fc7c0141670 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.065+0000 7fc7c4c96640 1 librados: init done 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.067+0000 7fc7c4c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.067+0000 7fc7c4c96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.067+0000 7fc7c4c96640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.067+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c01453d0 con 0x7fc7c0141670 2024-03-30T07:16:56.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.085+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7b0003970 con 0x7fc7c0141670 2024-03-30T07:16:56.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.085+0000 7fc7aeffd640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 0x7fc7a006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.085+0000 7fc7be59c640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 0x7fc7a006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.085+0000 7fc7be59c640 10 monclient: get_auth_request con 0x7fc7a006c5e0 auth_method 0 2024-03-30T07:16:56.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.086+0000 7fc7aeffd640 10 client.4393.objecter ms_handle_connect 0x7fc7a006c5e0 2024-03-30T07:16:56.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.086+0000 7fc7be59c640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 0x7fc7a006eaa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8005880 tx=0x7fc7b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.089+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc7b0057a00 con 0x7fc7c0141670 2024-03-30T07:16:56.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.089+0000 7fc7aeffd640 10 client.4393.objecter ms_dispatch 0x7fc7c0000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:56.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.089+0000 7fc7aeffd640 3 client.4393.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:56.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.089+0000 7fc7aeffd640 3 client.4393.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:56.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.089+0000 7fc7aeffd640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-30T07:16:56.146 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T07:16:56.189 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.250+0000 7f7f68d00640 10 monclient: get_monmap_and_config 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.250+0000 7f7f68d00640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.250+0000 7f7f68d00640 10 monclient: monmap: 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:56.251251+0000 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:56.251251+0000 2024-03-30T07:16:56.251 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:56.252 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 Processor -- start 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 -- start start 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient: init 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f640b0120 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f640a84a0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7f64144ab0 0x7f7f6414ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-a con 0x7f7f64144ab0 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f640a8900 con 0x7f7f640a84a0 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f640a8bd0 con 0x7f7f640b0120 2024-03-30T07:16:56.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f640b1430 con 0x7f7f64144ab0 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.251+0000 7f7f68d00640 10 monclient(hunting): authenticate will time out at 1542.367035s 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.252+0000 7f7f62d9d640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7f64144ab0 0x7f7f6414ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.252+0000 7f7f61d9b640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7f640a84a0 0x7f7f640b0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40902/0 (socket says 172.21.15.47:40902) 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.252+0000 7f7f61d9b640 1 -- 172.21.15.47:0/3566903733 learned_addr learned my addr 172.21.15.47:0/3566903733 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.253+0000 7f7f62d9d640 10 monclient(hunting): get_auth_request con 0x7f7f64144ab0 auth_method 0 2024-03-30T07:16:56.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.253+0000 7f7f62d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.253+0000 7f7f62d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.253+0000 7f7f62d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.254+0000 7f7f62d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.254+0000 7f7f62d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.254+0000 7f7f62d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 1 -- 172.21.15.47:0/3566903733 >> v1:172.21.15.52:6789/0 conn(0x7f7f640b0120 legacy=0x7f7f640712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 1 -- 172.21.15.47:0/3566903733 >> v1:172.21.15.47:6789/0 conn(0x7f7f640a84a0 legacy=0x7f7f640b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 1 monclient: found mon.noname-a 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.256+0000 7f7f62d9d640 1 -- 172.21.15.47:0/3566903733 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64144f00 con 0x7f7f64144ab0 2024-03-30T07:16:56.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.261+0000 7f7f62d9d640 1 --2- 172.21.15.47:0/3566903733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f64144ab0 0x7f7f6414ce30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009d90 tx=0x7f7f5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e092ca2d07e73080 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 1 -- 172.21.15.47:0/3566903733 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c02cc60 con 0x7f7f64144ab0 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: dump: 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.266 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 1 -- 172.21.15.47:0/3566903733 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c02cdc0 con 0x7f7f64144ab0 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 1 -- 172.21.15.47:0/3566903733 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c034660 con 0x7f7f64144ab0 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f68d00640 5 monclient: authenticate success, global_id 4350 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f68d00640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f68d00640 4 set_mon_vals no callback set 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.265+0000 7f7f6159a640 10 monclient: dump: 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.267 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.268 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 10 monclient: shutdown 2024-03-30T07:16:56.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 -- 172.21.15.47:0/3566903733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f64144ab0 msgr2=0x7f7f6414ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 --2- 172.21.15.47:0/3566903733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f64144ab0 0x7f7f6414ce30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009d90 tx=0x7f7f5c02beb0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 -- 172.21.15.47:0/3566903733 shutdown_connections 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 --2- 172.21.15.47:0/3566903733 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f64144ab0 0x7f7f6414ce30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 -- 172.21.15.47:0/3566903733 >> 172.21.15.47:0/3566903733 conn(0x7f7f640b70b0 msgr2=0x7f7f640b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 -- 172.21.15.47:0/3566903733 shutdown_connections 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.266+0000 7f7f68d00640 1 -- 172.21.15.47:0/3566903733 wait complete. 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 10 monclient: monmap: 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.271 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 librados: starting msgr at 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 librados: starting objecter 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 Processor -- start 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 -- start start 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 librados: setting wanted keys 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 1 librados: calling monclient init 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.267+0000 7f7f68d00640 10 monclient: init 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-a con 0x7f7f640a84a0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f640b0120 0x7f7f640ac300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-c con 0x7f7f640b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7f64144ab0 0x7f7f6413a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): picked mon.noname-b con 0x7f7f64144ab0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640ad240 con 0x7f7f640a84a0 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640acc60 con 0x7f7f64144ab0 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f640ac000 con 0x7f7f640b0120 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f68d00640 10 monclient(hunting): authenticate will time out at 1542.383540s 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.268+0000 7f7f6259c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f640b0120 0x7f7f640ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51290/0 (socket says 172.21.15.47:51290) 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 1 -- 172.21.15.47:0/215764620 learned_addr learned my addr 172.21.15.47:0/215764620 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 10 monclient(hunting): get_auth_request con 0x7f7f640a84a0 auth_method 0 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f6259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f61d9b640 10 monclient(hunting): get_auth_request con 0x7f7f640b0120 auth_method 0 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f61d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.269+0000 7f7f61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.270+0000 7f7f61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 -- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f640b0120 msgr2=0x7f7f640ac300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f640b0120 0x7f7f640ac300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 -- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7f64144ab0 msgr2=0x7f7f6413a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7f64144ab0 0x7f7f6413a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 monclient: found mon.noname-a 2024-03-30T07:16:56.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.274+0000 7f7f6259c640 1 -- 172.21.15.47:0/215764620 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c009a70 con 0x7f7f640a84a0 2024-03-30T07:16:56.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.277+0000 7f7f6259c640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c410 tx=0x7f7f5400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.277+0000 7f7f52ffd640 10 client.?.objecter ms_handle_connect 0x7f7f640a84a0 2024-03-30T07:16:56.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.277+0000 7f7f52ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540035f0 con 0x7f7f640a84a0 2024-03-30T07:16:56.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: dump: 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54003770 con 0x7f7f640a84a0 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 5 monclient: authenticate success, global_id 4356 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 10 monclient: _renew_subs 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f640ac840 con 0x7f7f640a84a0 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54003b20 con 0x7f7f640a84a0 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f52ffd640 10 monclient: dump: 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.284 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f60d99640 4 set_mon_vals no callback set 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 10 monclient: _renew_subs 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6413a780 con 0x7f7f640a84a0 2024-03-30T07:16:56.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.281+0000 7f7f68d00640 1 librados: init done 2024-03-30T07:16:56.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.282+0000 7f7f68d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.282+0000 7f7f68d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.282+0000 7f7f68d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.282+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6404e030 con 0x7f7f640a84a0 2024-03-30T07:16:56.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.296+0000 7f7980b27640 10 monclient: get_monmap_and_config 2024-03-30T07:16:56.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.296+0000 7f7980b27640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.296+0000 7f7980b27640 10 monclient: monmap: 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:56.297560+0000 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:56.297560+0000 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.298 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:56.299 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.297+0000 7f7980b27640 1 Processor -- start 2024-03-30T07:16:56.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.297+0000 7f7980b27640 1 -- start start 2024-03-30T07:16:56.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.297+0000 7f7980b27640 10 monclient: init 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f797c0adad0 0x7f797c13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-c con 0x7f797c0adad0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f797c0adeb0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f797c06e550 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c13e310 con 0x7f797c0adeb0 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797c06eb90 con 0x7f797c06e550 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797c070a10 con 0x7f797c0adad0 2024-03-30T07:16:56.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.298+0000 7f7980b27640 10 monclient(hunting): authenticate will time out at 1542.413387s 2024-03-30T07:16:56.317 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T07:16:56.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.349+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fc7b0053d40 con 0x7fc7c0141670 2024-03-30T07:16:56.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.349+0000 7fc7aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.349+0000 7fc7aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:56.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.353+0000 7f797a59c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f797c0adad0 0x7f797c13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f797c0adad0 0x7f797c13ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44400/0 (socket says 172.21.15.47:44400) 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 1 -- 172.21.15.47:0/735138909 learned_addr learned my addr 172.21.15.47:0/735138909 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 10 monclient(hunting): get_auth_request con 0x7f797c0adad0 auth_method 0 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.354+0000 7f797a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.355+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.355+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.355+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 10 monclient(hunting): handle_auth_done global_id 4396 payload 274 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 1 -- 172.21.15.47:0/735138909 >> v1:172.21.15.52:6789/0 conn(0x7f797c06e550 legacy=0x7f797c148ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 1 -- 172.21.15.47:0/735138909 >> v1:172.21.15.47:6789/0 conn(0x7f797c0adeb0 legacy=0x7f797c071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 1 monclient: found mon.noname-c 2024-03-30T07:16:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.361+0000 7f797a59c640 1 -- 172.21.15.47:0/735138909 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c070cf0 con 0x7f797c0adad0 2024-03-30T07:16:56.366 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T07:16:56.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.369+0000 7f797a59c640 1 --2- 172.21.15.47:0/735138909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c0adad0 0x7f797c13ebb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f797400d150 tx=0x7f797402e060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96d2161a1a0c17c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 1 -- 172.21.15.47:0/735138909 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797402ea00 con 0x7f797c0adad0 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: dump: 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.378 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 1 -- 172.21.15.47:0/735138909 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797402eb60 con 0x7f797c0adad0 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 1 -- 172.21.15.47:0/735138909 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974009600 con 0x7f797c0adad0 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.374+0000 7f797959a640 10 monclient: dump: 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.375+0000 7f7980b27640 5 monclient: authenticate success, global_id 4396 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.375+0000 7f7980b27640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.376+0000 7f7980b27640 4 set_mon_vals no callback set 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.376+0000 7f7980b27640 10 monclient: shutdown 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.377+0000 7f7980b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.377+0000 7f7980b27640 1 -- 172.21.15.47:0/735138909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c0adad0 msgr2=0x7f797c13ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.377+0000 7f7980b27640 1 --2- 172.21.15.47:0/735138909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c0adad0 0x7f797c13ebb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f797400d150 tx=0x7f797402e060 comp rx=0 tx=0).stop 2024-03-30T07:16:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.378+0000 7f7980b27640 1 -- 172.21.15.47:0/735138909 shutdown_connections 2024-03-30T07:16:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.378+0000 7f7980b27640 1 --2- 172.21.15.47:0/735138909 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c0adad0 0x7f797c13ebb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.378+0000 7f7980b27640 1 -- 172.21.15.47:0/735138909 >> 172.21.15.47:0/735138909 conn(0x7f797c0b30d0 msgr2=0x7f797c0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.378+0000 7f7980b27640 1 -- 172.21.15.47:0/735138909 shutdown_connections 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.378+0000 7f7980b27640 1 -- 172.21.15.47:0/735138909 wait complete. 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.379+0000 7f7980b27640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.379+0000 7f7980b27640 10 monclient: monmap: 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.383 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 librados: starting msgr at 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 librados: starting objecter 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 Processor -- start 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 -- start start 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 librados: setting wanted keys 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 1 librados: calling monclient init 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.380+0000 7f7980b27640 10 monclient: init 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-b con 0x7f797c06e550 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:56.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f797c0adad0 0x7f797c140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-a con 0x7f797c0adad0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f797c141420 0x7f797c1d9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): picked mon.noname-c con 0x7f797c141420 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c142340 con 0x7f797c0adad0 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c13c210 con 0x7f797c06e550 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797c1404f0 con 0x7f797c141420 2024-03-30T07:16:56.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f7980b27640 10 monclient(hunting): authenticate will time out at 1542.496428s 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f797a59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.381+0000 7f797ad9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f797c141420 0x7f797c1d9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:44410/0 (socket says 172.21.15.47:44410) 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 1 -- 172.21.15.47:0/2990590699 learned_addr learned my addr 172.21.15.47:0/2990590699 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): get_auth_request con 0x7f797c06e550 auth_method 0 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797ad9d640 10 monclient(hunting): get_auth_request con 0x7f797c141420 auth_method 0 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.382+0000 7f797ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.383+0000 7f797ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.383+0000 7f797ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.383+0000 7f797ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 -- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f797c141420 msgr2=0x7f797c1d9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f797c141420 0x7f797c1d9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 -- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f797c0adad0 msgr2=0x7f797c140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f797c0adad0 0x7f797c140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 monclient: found mon.noname-b 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.386+0000 7f797a59c640 1 -- 172.21.15.47:0/2990590699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797400bd40 con 0x7f797c06e550 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.389+0000 7f797a59c640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f80 tx=0x7f7974004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.389+0000 7f7972ffd640 10 client.?.objecter ms_handle_connect 0x7f797c06e550 2024-03-30T07:16:56.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.389+0000 7f7972ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797402ea00 con 0x7f797c06e550 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: dump: 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.394 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974036820 con 0x7f797c06e550 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7972ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.393+0000 7f7978d99640 4 set_mon_vals no callback set 2024-03-30T07:16:56.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 5 monclient: authenticate success, global_id 4402 2024-03-30T07:16:56.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974036bd0 con 0x7f797c06e550 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 10 monclient: _renew_subs 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c13f270 con 0x7f797c06e550 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7972ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7972ffd640 10 monclient: dump: 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.397 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 10 monclient: _renew_subs 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c13dee0 con 0x7f797c06e550 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.394+0000 7f7980b27640 1 librados: init done 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.395+0000 7f7980b27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.395+0000 7f7980b27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.395+0000 7f7980b27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.395+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c04e030 con 0x7f797c06e550 2024-03-30T07:16:56.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.409+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7f54003c80 con 0x7f7f640a84a0 2024-03-30T07:16:56.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.409+0000 7f7f52ffd640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 0x7f7f4406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.409+0000 7f7f61d9b640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 0x7f7f4406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.409+0000 7f7f61d9b640 10 monclient: get_auth_request con 0x7f7f4406c5e0 auth_method 0 2024-03-30T07:16:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.410+0000 7f7f52ffd640 10 client.4356.objecter ms_handle_connect 0x7f7f4406c5e0 2024-03-30T07:16:56.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.410+0000 7f7f61d9b640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 0x7f7f4406eaa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f64069620 tx=0x7f7f58007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.414+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7974036d30 con 0x7f797c06e550 2024-03-30T07:16:56.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.415+0000 7f7972ffd640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 0x7f795c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.415+0000 7f7979d9b640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 0x7f795c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.415+0000 7f7979d9b640 10 monclient: get_auth_request con 0x7f795c06c5e0 auth_method 0 2024-03-30T07:16:56.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.415+0000 7f7979d9b640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 0x7f795c06eaa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f797c0b0e80 tx=0x7f7964007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.415+0000 7f7972ffd640 10 client.4402.objecter ms_handle_connect 0x7f795c06c5e0 2024-03-30T07:16:56.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.417+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f797407c6e0 con 0x7f797c06e550 2024-03-30T07:16:56.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.417+0000 7f7972ffd640 10 client.4402.objecter ms_dispatch 0x7f797c000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:56.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.417+0000 7f7972ffd640 3 client.4402.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:56.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.417+0000 7f7972ffd640 3 client.4402.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:56.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.417+0000 7f7972ffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-30T07:16:56.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 10 monclient: get_monmap_and_config 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 10 monclient: monmap: 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:56.430926+0000 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:56.430926+0000 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:56.433 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 1 Processor -- start 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 1 -- start start 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.430+0000 7f48ad327640 10 monclient: init 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-b con 0x7f48a80b0120 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f48a8144ab0 0x7f48a80712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-c con 0x7f48a8144ab0 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48a80b0d20 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80a8bd0 con 0x7f48a80b0d20 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a80aa4f0 con 0x7f48a8144ab0 2024-03-30T07:16:56.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a8144f10 con 0x7f48a80b0120 2024-03-30T07:16:56.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.431+0000 7f48ad327640 10 monclient(hunting): authenticate will time out at 1542.546835s 2024-03-30T07:16:56.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.432+0000 7f48a6ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:55130/0 (socket says 172.21.15.47:55130) 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 1 -- 172.21.15.47:0/511130146 learned_addr learned my addr 172.21.15.47:0/511130146 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): get_auth_request con 0x7f48a80b0120 auth_method 0 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.433+0000 7f48a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 1 -- 172.21.15.47:0/511130146 >> v2:172.21.15.52:3300/0 conn(0x7f48a8144ab0 msgr2=0x7f48a80712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 1 --2- 172.21.15.47:0/511130146 >> v2:172.21.15.52:3300/0 conn(0x7f48a8144ab0 0x7f48a80712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 1 -- 172.21.15.47:0/511130146 >> v1:172.21.15.47:6789/0 conn(0x7f48a80b0d20 legacy=0x7f48a81415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 1 monclient: found mon.noname-b 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.436+0000 7f48a6ffd640 1 -- 172.21.15.47:0/511130146 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a80b3780 con 0x7f48a80b0120 2024-03-30T07:16:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.439+0000 7f48a6ffd640 1 --2- 172.21.15.47:0/511130146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4898009870 tx=0x7f489802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50af7ec4a09a8ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 1 -- 172.21.15.47:0/511130146 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898031640 con 0x7f48a80b0120 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: dump: 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 1 -- 172.21.15.47:0/511130146 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48980317a0 con 0x7f48a80b0120 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 1 -- 172.21.15.47:0/511130146 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4898031b30 con 0x7f48a80b0120 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48a5ffb640 10 monclient: dump: 2024-03-30T07:16:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48ad327640 5 monclient: authenticate success, global_id 4298 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48ad327640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.443+0000 7f48ad327640 4 set_mon_vals no callback set 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 10 monclient: shutdown 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 -- 172.21.15.47:0/511130146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 msgr2=0x7f48a80a84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 --2- 172.21.15.47:0/511130146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4898009870 tx=0x7f489802b8f0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 -- 172.21.15.47:0/511130146 shutdown_connections 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 --2- 172.21.15.47:0/511130146 >> v2:172.21.15.52:3300/0 conn(0x7f48a8144ab0 0x7f48a80712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 --2- 172.21.15.47:0/511130146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0120 0x7f48a80a84a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 -- 172.21.15.47:0/511130146 >> 172.21.15.47:0/511130146 conn(0x7f48a80b70b0 msgr2=0x7f48a80b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 -- 172.21.15.47:0/511130146 shutdown_connections 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.444+0000 7f48ad327640 1 -- 172.21.15.47:0/511130146 wait complete. 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.445+0000 7f48ad327640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.445+0000 7f48ad327640 10 monclient: monmap: 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 librados: starting msgr at 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 librados: starting objecter 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 Processor -- start 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 -- start start 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 librados: setting wanted keys 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 1 librados: calling monclient init 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.446+0000 7f48ad327640 10 monclient: init 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:16:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48a80b0120 0x7f48a80adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-b con 0x7f48a80b0120 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-c con 0x7f48a80b0d20 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.447+0000 7f48ad327640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a8144ab0 0x7f48a81d97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): picked mon.noname-a con 0x7f48a8144ab0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a80ae450 con 0x7f48a8144ab0 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a81d9ce0 con 0x7f48a80b0120 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a80ae070 con 0x7f48a80b0d20 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48ad327640 10 monclient(hunting): authenticate will time out at 1542.563233s 2024-03-30T07:16:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:55138/0 (socket says 172.21.15.47:55138) 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 1 -- 172.21.15.47:0/824112729 learned_addr learned my addr 172.21.15.47:0/824112729 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 10 monclient(hunting): get_auth_request con 0x7f48a80b0d20 auth_method 0 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.448+0000 7f48a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.449+0000 7f48a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.449+0000 7f48a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.449+0000 7f48a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 -- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48a80b0120 msgr2=0x7f48a80adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48a80b0120 0x7f48a80adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 -- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a8144ab0 msgr2=0x7f48a81d97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a8144ab0 0x7f48a81d97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 monclient: found mon.noname-c 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.453+0000 7f48a67fc640 1 -- 172.21.15.47:0/824112729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48980094d0 con 0x7f48a80b0d20 2024-03-30T07:16:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.456+0000 7f48a67fc640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b410 tx=0x7f489c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.456+0000 7f488f7fe640 10 client.?.objecter ms_handle_connect 0x7f48a80b0d20 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.456+0000 7f488f7fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.458+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7f540010c0 con 0x7f7f640a84a0 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.458+0000 7f7f52ffd640 10 client.4356.objecter ms_dispatch 0x7f7f64000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.458+0000 7f7f52ffd640 3 client.4356.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.458+0000 7f7f52ffd640 3 client.4356.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.458+0000 7f7f52ffd640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-30T07:16:56.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c011830 con 0x7f48a80b0d20 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: dump: 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.461 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c0119b0 con 0x7f48a80b0d20 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 5 monclient: authenticate success, global_id 4301 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c011d60 con 0x7f48a80b0d20 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f488f7fe640 10 monclient: dump: 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.462 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 10 monclient: _renew_subs 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48a57fa640 4 set_mon_vals no callback set 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a8141d00 con 0x7f48a80b0d20 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 10 monclient: _renew_subs 2024-03-30T07:16:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a81421c0 con 0x7f48a80b0d20 2024-03-30T07:16:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.460+0000 7f48ad327640 1 librados: init done 2024-03-30T07:16:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.461+0000 7f48ad327640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.461+0000 7f48ad327640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.461+0000 7f48ad327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:56.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.461+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a80aef90 con 0x7f48a80b0d20 2024-03-30T07:16:56.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.481+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f489c01a400 con 0x7f48a80b0d20 2024-03-30T07:16:56.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.481+0000 7f488f7fe640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 0x7f489406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.481+0000 7f48a6ffd640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 0x7f489406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.481+0000 7f48a6ffd640 10 monclient: get_auth_request con 0x7f489406c610 auth_method 0 2024-03-30T07:16:56.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.482+0000 7f488f7fe640 10 client.4301.objecter ms_handle_connect 0x7f489406c610 2024-03-30T07:16:56.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.482+0000 7f48a6ffd640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 0x7f489406ead0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4898036000 tx=0x7f4898030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.488+0000 7fac8ada2640 10 monclient: get_monmap_and_config 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.488+0000 7fac8ada2640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.488+0000 7fac8ada2640 10 monclient: monmap: 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:56.489357+0000 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:56.489357+0000 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.489 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:56.490 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:56.490 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:56.490 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:56.490 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:56.490 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.489+0000 7fac8ada2640 1 Processor -- start 2024-03-30T07:16:56.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.489+0000 7fac8ada2640 1 -- start start 2024-03-30T07:16:56.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.489+0000 7fac8ada2640 10 monclient: init 2024-03-30T07:16:56.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac840a8710 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac840b3500 0x7fac840b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-a con 0x7fac840b3500 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fac84144910 0x7fac840afd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-c con 0x7fac84144910 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840a8af0 con 0x7fac840a8710 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8413f000 con 0x7fac840b3500 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac840b43c0 con 0x7fac84144910 2024-03-30T07:16:56.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.490+0000 7fac8ada2640 10 monclient(hunting): authenticate will time out at 1542.605572s 2024-03-30T07:16:56.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.565+0000 7fc7c4c96640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:56.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.565+0000 7fc7c4c96640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:56.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.565+0000 7fc7c4c96640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.565+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc7c01453d0 con 0x7fc7c0141670 2024-03-30T07:16:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.606+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7f54053ff0 con 0x7f7f640a84a0 2024-03-30T07:16:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.606+0000 7f7f52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.606+0000 7f7f52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.624+0000 7fac8959f640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac840b3500 0x7fac840b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac840b3500 0x7fac840b3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51318/0 (socket says 172.21.15.47:51318) 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 1 -- 172.21.15.47:0/1133489749 learned_addr learned my addr 172.21.15.47:0/1133489749 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): get_auth_request con 0x7fac840b3500 auth_method 0 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.626+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 1 -- 172.21.15.47:0/1133489749 >> v2:172.21.15.52:3300/0 conn(0x7fac84144910 msgr2=0x7fac840afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 1 --2- 172.21.15.47:0/1133489749 >> v2:172.21.15.52:3300/0 conn(0x7fac84144910 0x7fac840afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 1 -- 172.21.15.47:0/1133489749 >> v1:172.21.15.52:6789/0 conn(0x7fac840a8710 legacy=0x7fac840b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 1 monclient: found mon.noname-a 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.629+0000 7fac8959f640 1 -- 172.21.15.47:0/1133489749 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840b14a0 con 0x7fac840b3500 2024-03-30T07:16:56.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.633+0000 7fac8959f640 1 --2- 172.21.15.47:0/1133489749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac840b3cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac840a9740 tx=0x7fac7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3c650eef2f58cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 1 -- 172.21.15.47:0/1133489749 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74031870 con 0x7fac840b3500 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: dump: 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.638 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 1 -- 172.21.15.47:0/1133489749 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac740319d0 con 0x7fac840b3500 2024-03-30T07:16:56.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac8ada2640 5 monclient: authenticate success, global_id 4359 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac8ada2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac8ada2640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 1 -- 172.21.15.47:0/1133489749 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74031d60 con 0x7fac840b3500 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac8ada2640 4 set_mon_vals no callback set 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.637+0000 7fac88d9e640 10 monclient: dump: 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.641 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 10 monclient: shutdown 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 -- 172.21.15.47:0/1133489749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 msgr2=0x7fac840b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 --2- 172.21.15.47:0/1133489749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac840b3cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fac840a9740 tx=0x7fac7402b910 comp rx=0 tx=0).stop 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 -- 172.21.15.47:0/1133489749 shutdown_connections 2024-03-30T07:16:56.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 --2- 172.21.15.47:0/1133489749 >> v2:172.21.15.52:3300/0 conn(0x7fac84144910 0x7fac840afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 --2- 172.21.15.47:0/1133489749 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac840b3cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 -- 172.21.15.47:0/1133489749 >> 172.21.15.47:0/1133489749 conn(0x7fac84065bc0 msgr2=0x7fac84065fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 -- 172.21.15.47:0/1133489749 shutdown_connections 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.638+0000 7fac8ada2640 1 -- 172.21.15.47:0/1133489749 wait complete. 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 10 monclient: build_initial_monmap 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 10 monclient: monmap: 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 librados: starting msgr at 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 librados: starting objecter 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 Processor -- start 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 -- start start 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 librados: setting wanted keys 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 1 librados: calling monclient init 2024-03-30T07:16:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.639+0000 7fac8ada2640 10 monclient: init 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac840a8710 0x7fac84141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-c con 0x7fac840a8710 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-a con 0x7fac840b3500 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac84144910 0x7fac8413d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): picked mon.noname-b con 0x7fac84144910 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8413c4c0 con 0x7fac840b3500 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8413d7a0 con 0x7fac84144910 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac84137010 con 0x7fac840a8710 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8ada2640 10 monclient(hunting): authenticate will time out at 1542.755670s 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.640+0000 7fac8959f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51324/0 (socket says 172.21.15.47:51324) 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 1 -- 172.21.15.47:0/244631064 learned_addr learned my addr 172.21.15.47:0/244631064 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): get_auth_request con 0x7fac840b3500 auth_method 0 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:56.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.641+0000 7fac8959f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:56.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-03-30T07:16:56.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 -- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac840a8710 msgr2=0x7fac84141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac840a8710 0x7fac84141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 -- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac84144910 msgr2=0x7fac8413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac84144910 0x7fac8413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 10 monclient: _finish_hunting 0 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 monclient: found mon.noname-a 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.644+0000 7fac8959f640 1 -- 172.21.15.47:0/244631064 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740094d0 con 0x7fac840b3500 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.647+0000 7fac8959f640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac74000c00 tx=0x7fac740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.647+0000 7fac7a7fc640 10 client.?.objecter ms_handle_connect 0x7fac840b3500 2024-03-30T07:16:56.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.647+0000 7fac7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7403ad10 con 0x7fac840b3500 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: dump: 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.652 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: _finish_auth 0 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7403ae70 con 0x7fac840b3500 2024-03-30T07:16:56.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 5 monclient: authenticate success, global_id 4362 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 10 monclient: _renew_subs 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac84137230 con 0x7fac840b3500 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74044610 con 0x7fac840b3500 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7a7fc640 10 monclient: dump: 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:56.654 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:56.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac7bfff640 4 set_mon_vals no callback set 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 10 monclient: _renew_subs 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac84137680 con 0x7fac840b3500 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.650+0000 7fac8ada2640 1 librados: init done 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.651+0000 7fac8ada2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.651+0000 7fac8ada2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.651+0000 7fac8ada2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.651+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8404e030 con 0x7fac840b3500 2024-03-30T07:16:56.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.672+0000 7fc7aeffd640 1 -- 172.21.15.47:0/576717488 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc7c01453d0 con 0x7fc7c0141670 2024-03-30T07:16:56.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.672+0000 7fc7aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-30T07:16:56.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.672+0000 7fc7aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:56.673 INFO:teuthology.orchestra.run.smithi047.stdout:60129542150 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 10 librados: watch_flush enter 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 10 librados: watch_flush exit 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 10 client.4393.objecter successfully canceled tick 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 msgr2=0x7fc7a006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 0x7fc7a006eaa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8005880 tx=0x7fc7b8009040 comp rx=0 tx=0).stop 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 10 monclient: shutdown 2024-03-30T07:16:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 msgr2=0x7fc7c01d6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000c390 tx=0x7fc7b000c850 comp rx=0 tx=0).stop 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 shutdown_connections 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fc7a006c5e0 0x7fc7a006eaa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fc7c0141670 0x7fc7c01d6ff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc7c013e930 0x7fc7c0144e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 --2- 172.21.15.47:0/576717488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc7c00b4280 0x7fc7c01456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 >> 172.21.15.47:0/576717488 conn(0x7fc7c0065e00 msgr2=0x7fc7c012e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 shutdown_connections 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.675+0000 7fc7c4c96640 1 -- 172.21.15.47:0/576717488 wait complete. 2024-03-30T07:16:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.676+0000 7fc7c4c96640 1 librados: shutdown 2024-03-30T07:16:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.688+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7974078320 con 0x7f797c06e550 2024-03-30T07:16:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.688+0000 7f7972ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.688+0000 7f7972ffd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:56.690 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.3 2024-03-30T07:16:56.690 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f489c059060 con 0x7f48a80b0d20 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 10 client.4301.objecter ms_dispatch 0x7f48a8000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 3 client.4301.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 3 client.4301.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.817+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f489c055430 con 0x7f48a80b0d20 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.818+0000 7f488f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.818+0000 7f488f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:56.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.865+0000 7f7f68d00640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:56.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.865+0000 7f7f68d00640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:56.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.865+0000 7f7f68d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:56.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.865+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7f640acdd0 con 0x7f7f640a84a0 2024-03-30T07:16:56.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.885+0000 7f7980b27640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:56.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.885+0000 7f7980b27640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:56.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.885+0000 7f7980b27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:56.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.885+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f797c04e030 con 0x7f797c06e550 2024-03-30T07:16:56.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.889+0000 7f7972ffd640 1 -- 172.21.15.47:0/2990590699 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f797407b940 con 0x7f797c06e550 2024-03-30T07:16:56.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.889+0000 7f7972ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-30T07:16:56.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.889+0000 7f7972ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:56.890 INFO:teuthology.orchestra.run.smithi047.stdout:60129542152 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 10 librados: watch_flush enter 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 10 librados: watch_flush exit 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 10 client.4402.objecter successfully canceled tick 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 msgr2=0x7f795c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 0x7f795c06eaa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f797c0b0e80 tx=0x7f7964007990 comp rx=0 tx=0).stop 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 10 monclient: shutdown 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 msgr2=0x7f797c132390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7974000f80 tx=0x7f7974004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 shutdown_connections 2024-03-30T07:16:56.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f795c06c5e0 0x7f795c06eaa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f797c141420 0x7f797c1d9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f797c0adad0 0x7f797c140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 --2- 172.21.15.47:0/2990590699 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f797c06e550 0x7f797c132390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 >> 172.21.15.47:0/2990590699 conn(0x7f797c0b30d0 msgr2=0x7f797c0710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 shutdown_connections 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 -- 172.21.15.47:0/2990590699 wait complete. 2024-03-30T07:16:56.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.891+0000 7f7980b27640 1 librados: shutdown 2024-03-30T07:16:56.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.5 2024-03-30T07:16:56.903 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:56.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.955+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fac7404d450 con 0x7fac840b3500 2024-03-30T07:16:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.956+0000 7fac7a7fc640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 0x7fac6806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.956+0000 7fac89da0640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 0x7fac6806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.956+0000 7fac89da0640 10 monclient: get_auth_request con 0x7fac6806c610 auth_method 0 2024-03-30T07:16:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.956+0000 7fac89da0640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 0x7fac6806ead0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fac80001560 tx=0x7fac80007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.956+0000 7fac7a7fc640 10 client.4362.objecter ms_handle_connect 0x7fac6806c610 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fac7402d040 con 0x7fac840b3500 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 10 client.4362.objecter ms_dispatch 0x7fac84000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 3 client.4362.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 3 client.4362.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac74077130 con 0x7fac840b3500 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.971+0000 7fac7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.983+0000 7f7f52ffd640 1 -- 172.21.15.47:0/215764620 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f54017090 con 0x7f7f640a84a0 2024-03-30T07:16:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.983+0000 7f7f52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-30T07:16:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.983+0000 7f7f52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:56.984 INFO:teuthology.orchestra.run.smithi047.stdout:60129542153 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 10 librados: watch_flush enter 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 10 librados: watch_flush exit 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 10 client.4356.objecter successfully canceled tick 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 msgr2=0x7f7f4406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 0x7f7f4406eaa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f64069620 tx=0x7f7f58007e80 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 10 monclient: shutdown 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 msgr2=0x7f7f64142600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c410 tx=0x7f7f5400c8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 shutdown_connections 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f7f4406c5e0 0x7f7f4406eaa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f7f64144ab0 0x7f7f6413a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f640b0120 0x7f7f640ac300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 --2- 172.21.15.47:0/215764620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f640a84a0 0x7f7f64142600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 >> 172.21.15.47:0/215764620 conn(0x7f7f640b70b0 msgr2=0x7f7f6413d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 shutdown_connections 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 -- 172.21.15.47:0/215764620 wait complete. 2024-03-30T07:16:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:56.985+0000 7f7f68d00640 1 librados: shutdown 2024-03-30T07:16:56.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.6 2024-03-30T07:16:56.999 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:57.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.037+0000 7f48ad327640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:57.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.038+0000 7f48ad327640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:57.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.038+0000 7f48ad327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:16:57.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.038+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f48a804e030 con 0x7f48a80b0d20 2024-03-30T07:16:57.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.054+0000 7f488f7fe640 1 -- 172.21.15.47:0/824112729 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f489c015020 con 0x7f48a80b0d20 2024-03-30T07:16:57.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.054+0000 7f488f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-30T07:16:57.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.054+0000 7f488f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:57.056 INFO:teuthology.orchestra.run.smithi047.stdout:81604378633 2024-03-30T07:16:57.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 10 librados: watch_flush enter 2024-03-30T07:16:57.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 10 librados: watch_flush exit 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 10 client.4301.objecter successfully canceled tick 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 msgr2=0x7f489406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 0x7f489406ead0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4898036000 tx=0x7f4898030000 comp rx=0 tx=0).stop 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 10 monclient: shutdown 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 msgr2=0x7f48a80ae970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f489c00b410 tx=0x7f489c00b8d0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 shutdown_connections 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f489406c610 0x7f489406ead0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48a8144ab0 0x7f48a81d97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48a80b0d20 0x7f48a80ae970 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 --2- 172.21.15.47:0/824112729 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f48a80b0120 0x7f48a80adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 >> 172.21.15.47:0/824112729 conn(0x7f48a80b70b0 msgr2=0x7f48a8143240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 shutdown_connections 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 -- 172.21.15.47:0/824112729 wait complete. 2024-03-30T07:16:57.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.056+0000 7f48ad327640 1 librados: shutdown 2024-03-30T07:16:57.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378633 for osd.7 2024-03-30T07:16:57.069 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.176+0000 7fac8ada2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.176+0000 7fac8ada2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.176+0000 7fac8ada2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:57.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.176+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fac84069ce0 con 0x7fac840b3500 2024-03-30T07:16:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.181+0000 7fac7a7fc640 1 -- 172.21.15.47:0/244631064 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac7407a750 con 0x7fac840b3500 2024-03-30T07:16:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.181+0000 7fac7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-30T07:16:57.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.181+0000 7fac7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:57.182 INFO:teuthology.orchestra.run.smithi047.stdout:68719476745 2024-03-30T07:16:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 10 librados: watch_flush enter 2024-03-30T07:16:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 10 librados: watch_flush exit 2024-03-30T07:16:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-30T07:16:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 10 client.4362.objecter successfully canceled tick 2024-03-30T07:16:57.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 msgr2=0x7fac6806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.183+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 0x7fac6806ead0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fac80001560 tx=0x7fac80007990 comp rx=0 tx=0).stop 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 10 monclient: shutdown 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 msgr2=0x7fac8413cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac74000c00 tx=0x7fac740046c0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 shutdown_connections 2024-03-30T07:16:57.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac6806c610 0x7fac6806ead0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac84144910 0x7fac8413d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac840b3500 0x7fac8413cb50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 --2- 172.21.15.47:0/244631064 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac840a8710 0x7fac84141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 >> 172.21.15.47:0/244631064 conn(0x7fac84065bc0 msgr2=0x7fac84143ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 shutdown_connections 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 -- 172.21.15.47:0/244631064 wait complete. 2024-03-30T07:16:57.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.184+0000 7fac8ada2640 1 librados: shutdown 2024-03-30T07:16:57.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476745 for osd.1 2024-03-30T07:16:57.197 DEBUG:teuthology.parallel:result is None 2024-03-30T07:16:57.197 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T07:16:57.198 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:16:57.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 10 monclient: get_monmap_and_config 2024-03-30T07:16:57.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 10 monclient: build_initial_monmap 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 10 monclient: monmap: 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:57.297761+0000 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:57.297761+0000 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:57.299 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:57.300 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 1 Processor -- start 2024-03-30T07:16:57.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 1 -- start start 2024-03-30T07:16:57.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.297+0000 7faccb8aa640 10 monclient: init 2024-03-30T07:16:57.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:57.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-30T07:16:57.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc40b4280 addr v1:172.21.15.52:6789/0 2024-03-30T07:16:57.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 0x7facc413eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-a con 0x7facc413e930 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:57.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7facc4141670 0x7facc4141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-c con 0x7facc4141670 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc413f2e0 con 0x7facc40b4280 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40b4660 con 0x7facc413e930 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40aa3c0 con 0x7facc4141670 2024-03-30T07:16:57.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7faccb8aa640 10 monclient(hunting): authenticate will time out at 1543.413532s 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.298+0000 7facc8e45640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 0x7facc413eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc9e47640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7facc4141670 0x7facc4141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 0x7facc413eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49544/0 (socket says 172.21.15.47:49544) 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 1 -- 172.21.15.47:0/3443345772 learned_addr learned my addr 172.21.15.47:0/3443345772 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:57.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): get_auth_request con 0x7facc413e930 auth_method 0 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc9e47640 10 monclient(hunting): get_auth_request con 0x7facc4141670 auth_method 0 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc9e47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc9e47640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc9e47640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.299+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.300+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.300+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.300+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 1 -- 172.21.15.47:0/3443345772 >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 msgr2=0x7facc413eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 1 --2- 172.21.15.47:0/3443345772 >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 0x7facc413eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 1 -- 172.21.15.47:0/3443345772 >> v1:172.21.15.52:6789/0 conn(0x7facc40b4280 legacy=0x7facc40a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 10 monclient: _finish_hunting 0 2024-03-30T07:16:57.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 1 monclient: found mon.noname-c 2024-03-30T07:16:57.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.301+0000 7facc9e47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:57.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.302+0000 7facc9e47640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.302+0000 7facc9e47640 1 -- 172.21.15.47:0/3443345772 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40aa630 con 0x7facc4141670 2024-03-30T07:16:57.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.305+0000 7facc9e47640 1 --2- 172.21.15.47:0/3443345772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc4141670 0x7facc4141a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7facc40b39e0 tx=0x7facbc02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da8712e749bb5e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:57.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 1 -- 172.21.15.47:0/3443345772 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc02cbf0 con 0x7facc4141670 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: dump: 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.319 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: _finish_auth 0 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: _check_auth_tickets 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:57.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 1 -- 172.21.15.47:0/3443345772 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc02cd50 con 0x7facc4141670 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 1 -- 172.21.15.47:0/3443345772 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc034660 con 0x7facc4141670 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7facbbfff640 10 monclient: dump: 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.321 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7faccb8aa640 5 monclient: authenticate success, global_id 4405 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7faccb8aa640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.308+0000 7faccb8aa640 4 set_mon_vals no callback set 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 10 monclient: shutdown 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 -- 172.21.15.47:0/3443345772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc4141670 msgr2=0x7facc4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 --2- 172.21.15.47:0/3443345772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc4141670 0x7facc4141a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7facc40b39e0 tx=0x7facbc02be40 comp rx=0 tx=0).stop 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 -- 172.21.15.47:0/3443345772 shutdown_connections 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 --2- 172.21.15.47:0/3443345772 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc4141670 0x7facc4141a80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 --2- 172.21.15.47:0/3443345772 >> v2:172.21.15.47:3300/0 conn(0x7facc413e930 0x7facc413eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 -- 172.21.15.47:0/3443345772 >> 172.21.15.47:0/3443345772 conn(0x7facc4065e00 msgr2=0x7facc4066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 -- 172.21.15.47:0/3443345772 shutdown_connections 2024-03-30T07:16:57.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.309+0000 7faccb8aa640 1 -- 172.21.15.47:0/3443345772 wait complete. 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 10 monclient: build_initial_monmap 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 10 monclient: monmap: 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 librados: starting msgr at 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 librados: starting objecter 2024-03-30T07:16:57.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 Processor -- start 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 -- start start 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 librados: setting wanted keys 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 1 librados: calling monclient init 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.310+0000 7faccb8aa640 10 monclient: init 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 0x7facc4140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-c con 0x7facc40b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 0x7facc41d9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-b con 0x7facc413e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc4141670 0x7facc412c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): picked mon.noname-a con 0x7facc4141670 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41317f0 con 0x7facc4141670 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41da540 con 0x7facc413e930 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc4132080 con 0x7facc40b4280 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7faccb8aa640 10 monclient(hunting): authenticate will time out at 1543.426506s 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7facc8e45640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 0x7facc41d9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.311+0000 7facc9646640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 0x7facc4140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc4141670 0x7facc412c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 0x7facc4140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:48010/0 (socket says 172.21.15.47:48010) 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 1 -- 172.21.15.47:0/888417333 learned_addr learned my addr 172.21.15.47:0/888417333 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): get_auth_request con 0x7facc40b4280 auth_method 0 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): get_auth_request con 0x7facc413e930 auth_method 0 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): get_auth_request con 0x7facc4141670 auth_method 0 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc8e45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9646640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.312+0000 7facc9e47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 -- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 msgr2=0x7facc4140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 0x7facc4140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 -- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc4141670 msgr2=0x7facc412c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc4141670 0x7facc412c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 10 monclient: _finish_hunting 0 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 monclient: found mon.noname-b 2024-03-30T07:16:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:57.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.315+0000 7facc8e45640 1 -- 172.21.15.47:0/888417333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009a70 con 0x7facc413e930 2024-03-30T07:16:57.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.317+0000 7facc8e45640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 0x7facc41d9dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facb400e850 tx=0x7facb400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:57.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.317+0000 7facb9ffb640 10 client.?.objecter ms_handle_connect 0x7facc413e930 2024-03-30T07:16:57.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.317+0000 7facb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:57.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb400c940 con 0x7facc413e930 2024-03-30T07:16:57.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: dump: 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb400caa0 con 0x7facc413e930 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb400ce10 con 0x7facc413e930 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 5 monclient: authenticate success, global_id 4408 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 10 monclient: _renew_subs 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc4132360 con 0x7facc413e930 2024-03-30T07:16:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facbb7fe640 4 set_mon_vals no callback set 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7facb9ffb640 10 monclient: dump: 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 10 monclient: _renew_subs 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc412c740 con 0x7facc413e930 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.321+0000 7faccb8aa640 1 librados: init done 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.322+0000 7faccb8aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.322+0000 7faccb8aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.322+0000 7faccb8aa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.322+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc41da310 con 0x7facc413e930 2024-03-30T07:16:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.343+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7facb4013470 con 0x7facc413e930 2024-03-30T07:16:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.343+0000 7facb9ffb640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 0x7faca806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.343+0000 7facc9646640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 0x7faca806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.343+0000 7facc9646640 10 monclient: get_auth_request con 0x7faca806c590 auth_method 0 2024-03-30T07:16:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.344+0000 7facc9646640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 0x7faca806ea50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7facc0000f60 tx=0x7facc0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.344+0000 7facb9ffb640 10 client.4408.objecter ms_handle_connect 0x7faca806c590 2024-03-30T07:16:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.345+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7facb40589d0 con 0x7facc413e930 2024-03-30T07:16:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.345+0000 7facb9ffb640 10 client.4408.objecter ms_dispatch 0x7facc4000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.345+0000 7facb9ffb640 3 client.4408.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.345+0000 7facb9ffb640 3 client.4408.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.345+0000 7facb9ffb640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-30T07:16:57.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.609+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7facb4054d10 con 0x7facc413e930 2024-03-30T07:16:57.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.609+0000 7facb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:57.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.609+0000 7facb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:57.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.791+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facc41da310 con 0x7faca806c590 2024-03-30T07:16:57.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.793+0000 7facb9ffb640 1 -- 172.21.15.47:0/888417333 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48086 (secure 0 0 0) 0x7facc41da310 con 0x7faca806c590 2024-03-30T07:16:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:57.795 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 10 librados: watch_flush enter 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 10 librados: watch_flush exit 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 10 client.4408.objecter successfully canceled tick 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 msgr2=0x7faca806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 0x7faca806ea50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7facc0000f60 tx=0x7facc0009040 comp rx=0 tx=0).stop 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 10 monclient: shutdown 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 msgr2=0x7facc41d9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 0x7facc41d9dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facb400e850 tx=0x7facb400ed10 comp rx=0 tx=0).stop 2024-03-30T07:16:57.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 shutdown_connections 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7faca806c590 0x7faca806ea50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7facc4141670 0x7facc412c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7facc413e930 0x7facc41d9dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 --2- 172.21.15.47:0/888417333 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7facc40b4280 0x7facc4140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 >> 172.21.15.47:0/888417333 conn(0x7facc4065e00 msgr2=0x7facc40b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 shutdown_connections 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 -- 172.21.15.47:0/888417333 wait complete. 2024-03-30T07:16:57.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.795+0000 7faccb8aa640 1 librados: shutdown 2024-03-30T07:16:57.808 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-03-30T07:16:55.930329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218956,"kb_used_data":2772,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755764,"statfs":{"total":773094113280,"available":772869902336,"internally_reserved":0,"allocated":2838528,"data_stored":1519588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.620826"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.155495+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:16:51.155495+0000","last_peered":"2024-03-30T07:16:51.155495+0000","last_clean":"2024-03-30T07:16:51.155495+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:16:51.155495+0000","last_undegraded":"2024-03-30T07:16:51.155495+0000","last_fullsized":"2024-03-30T07:16:51.155495+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:41:46.497967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.837885+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:16:51.837885+0000","last_peered":"2024-03-30T07:16:51.837885+0000","last_clean":"2024-03-30T07:16:51.837885+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:16:51.837885+0000","last_undegraded":"2024-03-30T07:16:51.837885+0000","last_fullsized":"2024-03-30T07:16:51.837885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.151091+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:16:51.151091+0000","last_peered":"2024-03-30T07:16:51.151091+0000","last_clean":"2024-03-30T07:16:51.151091+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:16:51.151091+0000","last_undegraded":"2024-03-30T07:16:51.151091+0000","last_fullsized":"2024-03-30T07:16:51.151091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:12.481693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.840922+0000","last_change":"2024-03-30T07:16:49.917154+0000","last_active":"2024-03-30T07:16:51.840922+0000","last_peered":"2024-03-30T07:16:51.840922+0000","last_clean":"2024-03-30T07:16:51.840922+0000","last_became_active":"2024-03-30T07:16:49.890030+0000","last_became_peered":"2024-03-30T07:16:49.890030+0000","last_unstale":"2024-03-30T07:16:51.840922+0000","last_undegraded":"2024-03-30T07:16:51.840922+0000","last_fullsized":"2024-03-30T07:16:51.840922+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:18:52.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.145603+0000","last_change":"2024-03-30T07:16:49.881449+0000","last_active":"2024-03-30T07:16:51.145603+0000","last_peered":"2024-03-30T07:16:51.145603+0000","last_clean":"2024-03-30T07:16:51.145603+0000","last_became_active":"2024-03-30T07:16:49.872403+0000","last_became_peered":"2024-03-30T07:16:49.872403+0000","last_unstale":"2024-03-30T07:16:51.145603+0000","last_undegraded":"2024-03-30T07:16:51.145603+0000","last_fullsized":"2024-03-30T07:16:51.145603+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:11:03.553087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.081305+0000","last_change":"2024-03-30T07:16:49.576511+0000","last_active":"2024-03-30T07:16:51.081305+0000","last_peered":"2024-03-30T07:16:51.081305+0000","last_clean":"2024-03-30T07:16:51.081305+0000","last_became_active":"2024-03-30T07:16:49.546956+0000","last_became_peered":"2024-03-30T07:16:49.546956+0000","last_unstale":"2024-03-30T07:16:51.081305+0000","last_undegraded":"2024-03-30T07:16:51.081305+0000","last_fullsized":"2024-03-30T07:16:51.081305+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:59:02.145807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.139546+0000","last_change":"2024-03-30T07:16:49.643848+0000","last_active":"2024-03-30T07:16:51.139546+0000","last_peered":"2024-03-30T07:16:51.139546+0000","last_clean":"2024-03-30T07:16:51.139546+0000","last_became_active":"2024-03-30T07:16:49.615316+0000","last_became_peered":"2024-03-30T07:16:49.615316+0000","last_unstale":"2024-03-30T07:16:51.139546+0000","last_undegraded":"2024-03-30T07:16:51.139546+0000","last_fullsized":"2024-03-30T07:16:51.139546+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:04:17.548941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.148438+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:16:51.148438+0000","last_peered":"2024-03-30T07:16:51.148438+0000","last_clean":"2024-03-30T07:16:51.148438+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:16:51.148438+0000","last_undegraded":"2024-03-30T07:16:51.148438+0000","last_fullsized":"2024-03-30T07:16:51.148438+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:47:35.848282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":122,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.125623+0000","last_change":"2024-03-30T07:16:36.238101+0000","last_active":"2024-03-30T07:16:51.125623+0000","last_peered":"2024-03-30T07:16:51.125623+0000","last_clean":"2024-03-30T07:16:51.125623+0000","last_became_active":"2024-03-30T07:16:36.202467+0000","last_became_peered":"2024-03-30T07:16:36.202467+0000","last_unstale":"2024-03-30T07:16:51.125623+0000","last_undegraded":"2024-03-30T07:16:51.125623+0000","last_fullsized":"2024-03-30T07:16:51.125623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_clean_scrub_stamp":"2024-03-30T07:16:32.203026+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:28:01.075945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.818}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.449}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.881}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.037000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.374000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.595000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7010000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.493}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.002000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.316000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.818000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.888999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.385000000000002}]}]},{"osd":4,"up_from":20,"seq":85899345930,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":88333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.093}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.218999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.832999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.68}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9710000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5020000000000007}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.558999999999999}]}]},{"osd":7,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":88333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4320000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.946}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.951000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6709999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":53.942999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.088000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.125}]}]},{"osd":1,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":491921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.132999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.931999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.088000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.277000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.795999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.111000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":651264,"data_stored":486173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.649}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":34.463000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.619999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":104.095}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":39.789000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.802}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.975999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":88333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":86.338999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.132999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":88.950000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.726000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.178999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.151999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":88333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":28.286999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0650000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.030000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.196999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.456}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":40.912999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:16:57.810 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.912+0000 7fa5f535c640 10 monclient: get_monmap_and_config 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.912+0000 7fa5f535c640 10 monclient: build_initial_monmap 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.912+0000 7fa5f535c640 10 monclient: monmap: 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.913 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:57.913188+0000 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:57.913188+0000 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 Processor -- start 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 -- start start 2024-03-30T07:16:57.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient: init 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fa5f006a830 0x7fa5f006ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f006a830 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa5f006ba90 0x7fa5f013e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f006ba90 addr v2:172.21.15.47:3300/0 2024-03-30T07:16:57.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5f013ecd0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f00ad520 con 0x7fa5f013ecd0 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f00af400 con 0x7fa5f006ba90 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f013f120 con 0x7fa5f006a830 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.913+0000 7fa5f535c640 10 monclient(hunting): authenticate will time out at 1544.028771s 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5eeffd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fa5f006a830 0x7fa5f006ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5ee7fc640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa5f006ba90 0x7fa5f013e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5ef7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa5f013ecd0 0x7fa5f0141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47490/0 (socket says 172.21.15.47:47490) 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5ef7fe640 1 -- 172.21.15.47:0/848606482 learned_addr learned my addr 172.21.15.47:0/848606482 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5eeffd640 10 monclient(hunting): get_auth_request con 0x7fa5f006a830 auth_method 0 2024-03-30T07:16:57.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.914+0000 7fa5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa5f006ba90 auth_method 0 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.915+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_done global_id 4411 payload 274 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 1 -- 172.21.15.47:0/848606482 >> v2:172.21.15.47:3300/0 conn(0x7fa5f006ba90 msgr2=0x7fa5f013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 1 --2- 172.21.15.47:0/848606482 >> v2:172.21.15.47:3300/0 conn(0x7fa5f006ba90 0x7fa5f013e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 1 -- 172.21.15.47:0/848606482 >> v1:172.21.15.47:6789/0 conn(0x7fa5f013ecd0 legacy=0x7fa5f0141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 1 monclient: found mon.noname-c 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:16:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.917+0000 7fa5eeffd640 1 -- 172.21.15.47:0/848606482 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f0141ea0 con 0x7fa5f006a830 2024-03-30T07:16:57.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.919+0000 7fa5eeffd640 1 --2- 172.21.15.47:0/848606482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006a830 0x7fa5f006ac10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2a00 tx=0x7fa5e802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3def935f7fab3aef server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 1 -- 172.21.15.47:0/848606482 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e802cc80 con 0x7fa5f006a830 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: dump: 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.924 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: _finish_auth 0 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: _check_auth_tickets 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 1 -- 172.21.15.47:0/848606482 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e802cde0 con 0x7fa5f006a830 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 1 -- 172.21.15.47:0/848606482 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8034660 con 0x7fa5f006a830 2024-03-30T07:16:57.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5f535c640 5 monclient: authenticate success, global_id 4411 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5f535c640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5f535c640 4 set_mon_vals no callback set 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.922+0000 7fa5edffb640 10 monclient: dump: 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.926 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 10 monclient: shutdown 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 -- 172.21.15.47:0/848606482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006a830 msgr2=0x7fa5f006ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 --2- 172.21.15.47:0/848606482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006a830 0x7fa5f006ac10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2a00 tx=0x7fa5e802bed0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 -- 172.21.15.47:0/848606482 shutdown_connections 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 --2- 172.21.15.47:0/848606482 >> v2:172.21.15.47:3300/0 conn(0x7fa5f006ba90 0x7fa5f013e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 --2- 172.21.15.47:0/848606482 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006a830 0x7fa5f006ac10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 -- 172.21.15.47:0/848606482 >> 172.21.15.47:0/848606482 conn(0x7fa5f00b5230 msgr2=0x7fa5f00b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 -- 172.21.15.47:0/848606482 shutdown_connections 2024-03-30T07:16:57.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.923+0000 7fa5f535c640 1 -- 172.21.15.47:0/848606482 wait complete. 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 10 monclient: build_initial_monmap 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 10 monclient: monmap: 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:57.930 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 librados: starting msgr at 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 librados: starting objecter 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 Processor -- start 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 -- start start 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 librados: setting wanted keys 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 1 librados: calling monclient init 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.924+0000 7fa5f535c640 10 monclient: init 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f006a830 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:57.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006ba90 0x7fa5f0134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f006ba90 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5f013ecd0 0x7fa5f01d9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f013ecd0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f01347f0 con 0x7fa5f006a830 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f01344f0 con 0x7fa5f006ba90 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0135310 con 0x7fa5f013ecd0 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5f535c640 10 monclient(hunting): authenticate will time out at 1544.040342s 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5ef7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5f013ecd0 0x7fa5f01d9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5ee7fc640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006ba90 0x7fa5f0134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:57.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:49572/0 (socket says 172.21.15.47:49572) 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 1 -- 172.21.15.47:0/3150139470 learned_addr learned my addr 172.21.15.47:0/3150139470 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 10 monclient(hunting): get_auth_request con 0x7fa5f006a830 auth_method 0 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.925+0000 7fa5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): get_auth_request con 0x7fa5f013ecd0 auth_method 0 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): get_auth_request con 0x7fa5f006ba90 auth_method 0 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.926+0000 7fa5ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 -- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5f013ecd0 msgr2=0x7fa5f01d9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5f013ecd0 0x7fa5f01d9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 -- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006ba90 msgr2=0x7fa5f0134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006ba90 0x7fa5f0134dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 10 monclient: _finish_hunting 0 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 monclient: found mon.noname-a 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:16:57.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.929+0000 7fa5eeffd640 1 -- 172.21.15.47:0/3150139470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8009a70 con 0x7fa5f006a830 2024-03-30T07:16:57.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.931+0000 7fa5eeffd640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa5e802c4e0 tx=0x7fa5e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:57.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.931+0000 7fa5d77fe640 10 client.?.objecter ms_handle_connect 0x7fa5f006a830 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.931+0000 7fa5d77fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e803e610 con 0x7fa5f006a830 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: dump: 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.937 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e803e790 con 0x7fa5f006a830 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5f535c640 5 monclient: authenticate success, global_id 4377 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e803eb40 con 0x7fa5f006a830 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5d77fe640 10 monclient: dump: 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:57.938 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5f535c640 10 monclient: _renew_subs 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5ed7fa640 4 set_mon_vals no callback set 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5f535c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.935+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f01424b0 con 0x7fa5f006a830 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 monclient: _renew_subs 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f01d9f50 con 0x7fa5f006a830 2024-03-30T07:16:57.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 1 librados: init done 2024-03-30T07:16:57.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:57.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:57.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:16:57.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:57.936+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f004e030 con 0x7fa5f006a830 2024-03-30T07:16:58.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.216+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa5e803b050 con 0x7fa5f006a830 2024-03-30T07:16:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.216+0000 7fa5d77fe640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 0x7fa5dc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.217+0000 7fa5ee7fc640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 0x7fa5dc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.217+0000 7fa5ee7fc640 10 monclient: get_auth_request con 0x7fa5dc06c5e0 auth_method 0 2024-03-30T07:16:58.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.217+0000 7fa5d77fe640 10 client.4377.objecter ms_handle_connect 0x7fa5dc06c5e0 2024-03-30T07:16:58.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.217+0000 7fa5ee7fc640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 0x7fa5dc06eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2030 tx=0x7fa5e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa5e807a910 con 0x7fa5f006a830 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 10 client.4377.objecter ms_dispatch 0x7fa5f0000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 3 client.4377.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 3 client.4377.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-30T07:16:58.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa5e8076c50 con 0x7fa5f006a830 2024-03-30T07:16:58.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:58.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.230+0000 7fa5d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:58.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.415+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5f006c110 con 0x7fa5dc06c5e0 2024-03-30T07:16:58.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.417+0000 7fa5d77fe640 1 -- 172.21.15.47:0/3150139470 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48144 (secure 0 0 0) 0x7fa5f006c110 con 0x7fa5dc06c5e0 2024-03-30T07:16:58.418 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:58.419 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:16:58.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 10 librados: watch_flush enter 2024-03-30T07:16:58.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 10 librados: watch_flush exit 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 10 client.4377.objecter successfully canceled tick 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 msgr2=0x7fa5dc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 0x7fa5dc06eaa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00b2030 tx=0x7fa5e4008040 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 10 monclient: shutdown 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 msgr2=0x7fa5f0136ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa5e802c4e0 tx=0x7fa5e8004a40 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 shutdown_connections 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa5dc06c5e0 0x7fa5dc06eaa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5f013ecd0 0x7fa5f01d9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa5f006ba90 0x7fa5f0134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 --2- 172.21.15.47:0/3150139470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5f006a830 0x7fa5f0136ab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 >> 172.21.15.47:0/3150139470 conn(0x7fa5f00b5230 msgr2=0x7fa5f00b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:58.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 shutdown_connections 2024-03-30T07:16:58.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 -- 172.21.15.47:0/3150139470 wait complete. 2024-03-30T07:16:58.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.419+0000 7fa5f535c640 1 librados: shutdown 2024-03-30T07:16:58.431 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-03-30T07:16:57.930783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219004,"kb_used_data":2832,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755716,"statfs":{"total":773094113280,"available":772869853184,"internally_reserved":0,"allocated":2899968,"data_stored":1548328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001584"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:52.095968+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:16:52.095968+0000","last_peered":"2024-03-30T07:16:52.095968+0000","last_clean":"2024-03-30T07:16:52.095968+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:16:52.095968+0000","last_undegraded":"2024-03-30T07:16:52.095968+0000","last_fullsized":"2024-03-30T07:16:52.095968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:41:46.497967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.837885+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:16:51.837885+0000","last_peered":"2024-03-30T07:16:51.837885+0000","last_clean":"2024-03-30T07:16:51.837885+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:16:51.837885+0000","last_undegraded":"2024-03-30T07:16:51.837885+0000","last_fullsized":"2024-03-30T07:16:51.837885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:52.094997+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:16:52.094997+0000","last_peered":"2024-03-30T07:16:52.094997+0000","last_clean":"2024-03-30T07:16:52.094997+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:16:52.094997+0000","last_undegraded":"2024-03-30T07:16:52.094997+0000","last_fullsized":"2024-03-30T07:16:52.094997+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:12.481693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.840922+0000","last_change":"2024-03-30T07:16:49.917154+0000","last_active":"2024-03-30T07:16:51.840922+0000","last_peered":"2024-03-30T07:16:51.840922+0000","last_clean":"2024-03-30T07:16:51.840922+0000","last_became_active":"2024-03-30T07:16:49.890030+0000","last_became_peered":"2024-03-30T07:16:49.890030+0000","last_unstale":"2024-03-30T07:16:51.840922+0000","last_undegraded":"2024-03-30T07:16:51.840922+0000","last_fullsized":"2024-03-30T07:16:51.840922+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:18:52.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:52.091788+0000","last_change":"2024-03-30T07:16:49.881449+0000","last_active":"2024-03-30T07:16:52.091788+0000","last_peered":"2024-03-30T07:16:52.091788+0000","last_clean":"2024-03-30T07:16:52.091788+0000","last_became_active":"2024-03-30T07:16:49.872403+0000","last_became_peered":"2024-03-30T07:16:49.872403+0000","last_unstale":"2024-03-30T07:16:52.091788+0000","last_undegraded":"2024-03-30T07:16:52.091788+0000","last_fullsized":"2024-03-30T07:16:52.091788+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:11:03.553087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.189559+0000","last_change":"2024-03-30T07:16:49.576511+0000","last_active":"2024-03-30T07:16:51.189559+0000","last_peered":"2024-03-30T07:16:51.189559+0000","last_clean":"2024-03-30T07:16:51.189559+0000","last_became_active":"2024-03-30T07:16:49.546956+0000","last_became_peered":"2024-03-30T07:16:49.546956+0000","last_unstale":"2024-03-30T07:16:51.189559+0000","last_undegraded":"2024-03-30T07:16:51.189559+0000","last_fullsized":"2024-03-30T07:16:51.189559+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:59:02.145807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:52.084692+0000","last_change":"2024-03-30T07:16:49.643848+0000","last_active":"2024-03-30T07:16:52.084692+0000","last_peered":"2024-03-30T07:16:52.084692+0000","last_clean":"2024-03-30T07:16:52.084692+0000","last_became_active":"2024-03-30T07:16:49.615316+0000","last_became_peered":"2024-03-30T07:16:49.615316+0000","last_unstale":"2024-03-30T07:16:52.084692+0000","last_undegraded":"2024-03-30T07:16:52.084692+0000","last_fullsized":"2024-03-30T07:16:52.084692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:04:17.548941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:52.093877+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:16:52.093877+0000","last_peered":"2024-03-30T07:16:52.093877+0000","last_clean":"2024-03-30T07:16:52.093877+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:16:52.093877+0000","last_undegraded":"2024-03-30T07:16:52.093877+0000","last_fullsized":"2024-03-30T07:16:52.093877+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:47:35.848282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":122,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T07:16:51.125623+0000","last_change":"2024-03-30T07:16:36.238101+0000","last_active":"2024-03-30T07:16:51.125623+0000","last_peered":"2024-03-30T07:16:51.125623+0000","last_clean":"2024-03-30T07:16:51.125623+0000","last_became_active":"2024-03-30T07:16:36.202467+0000","last_became_peered":"2024-03-30T07:16:36.202467+0000","last_unstale":"2024-03-30T07:16:51.125623+0000","last_undegraded":"2024-03-30T07:16:51.125623+0000","last_fullsized":"2024-03-30T07:16:51.125623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_clean_scrub_stamp":"2024-03-30T07:16:32.203026+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:28:01.075945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.818}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.449}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.2200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.881}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.037000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.374000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.595000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7010000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.493}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.002000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.316000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.818000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.888999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.385000000000002}]}]},{"osd":4,"up_from":20,"seq":85899345931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.5830000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.923}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.077999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2409999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.914999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.594999999999999}]}]},{"osd":7,"up_from":19,"seq":81604378634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.186}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2220000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.960000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.369}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.736000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8019999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.367000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476745,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":491921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.132999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.931999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.088000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.277000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":29.795999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.111000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":663552,"data_stored":491921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3419999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.509}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.659000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.257999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.606}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.103000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.562000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2390000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.190000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.920999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.037000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.861000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.547000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":94081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.627000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.513}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.015999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.835999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.396000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.182}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.411000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:16:58.432 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T07:16:58.432 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T07:16:58.433 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-30T07:16:58.433 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-30T07:16:58.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.531+0000 7efd26d29640 10 monclient: get_monmap_and_config 2024-03-30T07:16:58.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.531+0000 7efd26d29640 10 monclient: build_initial_monmap 2024-03-30T07:16:58.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.531+0000 7efd26d29640 10 monclient: monmap: 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:16:58.532435+0000 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:16:58.532435+0000 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:16:58.533 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 1 Processor -- start 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 1 -- start start 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 10 monclient: init 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.532+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd200719f0 addr v1:172.21.15.47:6789/0 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd200a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-b con 0x7efd200b0c70 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 0x7efd2013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-c con 0x7efd2013e800 addr v2:172.21.15.52:3300/0 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd200a8a00 con 0x7efd200719f0 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd20071dd0 con 0x7efd2013e800 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:58.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd20145130 con 0x7efd200b0c70 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd26d29640 10 monclient(hunting): authenticate will time out at 1544.648184s 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd24ac5640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efd200719f0 0x7efd20144a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47504/0 (socket says 172.21.15.47:47504) 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd24ac5640 1 -- 172.21.15.47:0/642819915 learned_addr learned my addr 172.21.15.47:0/642819915 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd17fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd200a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd252c6640 1 --2- 172.21.15.47:0/642819915 >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 0x7efd2013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd17fff640 10 monclient(hunting): get_auth_request con 0x7efd200b0c70 auth_method 0 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.533+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): get_auth_request con 0x7efd2013e800 auth_method 0 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:58.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.534+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 1 -- 172.21.15.47:0/642819915 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774083433 0 0) 0x7efd200a8a00 con 0x7efd200719f0 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 10 monclient(hunting): _negotiate 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 10 monclient(hunting): my global_id is 4380 2024-03-30T07:16:58.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd177fe640 1 -- 172.21.15.47:0/642819915 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd08003520 con 0x7efd200719f0 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 1 -- 172.21.15.47:0/642819915 >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 msgr2=0x7efd2013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 1 --2- 172.21.15.47:0/642819915 >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 0x7efd2013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 1 -- 172.21.15.47:0/642819915 >> v1:172.21.15.47:6789/0 conn(0x7efd200719f0 legacy=0x7efd20144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 10 monclient: _finish_hunting 0 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 1 monclient: found mon.noname-b 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd17fff640 1 -- 172.21.15.47:0/642819915 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd200aa070 con 0x7efd200b0c70 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.536+0000 7efd252c6640 1 --2- 172.21.15.47:0/642819915 >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 0x7efd2013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:16:58.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.538+0000 7efd17fff640 1 --2- 172.21.15.47:0/642819915 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd200a8e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd100097a0 tx=0x7efd1002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70e927ccf2c52452 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:58.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 1 -- 172.21.15.47:0/642819915 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1002f050 con 0x7efd200b0c70 2024-03-30T07:16:58.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:58.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:58.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: dump: 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: _finish_auth 0 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: _check_auth_tickets 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:58.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 1 -- 172.21.15.47:0/642819915 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd10031700 con 0x7efd200b0c70 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 5 monclient: authenticate success, global_id 4310 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 10 monclient: get_monmap_and_config success 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 4 set_mon_vals no callback set 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 1 -- 172.21.15.47:0/642819915 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10031a90 con 0x7efd200b0c70 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd177fe640 10 monclient: dump: 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.545 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 10 monclient: shutdown 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 -- 172.21.15.47:0/642819915 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 msgr2=0x7efd200a8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 --2- 172.21.15.47:0/642819915 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd200a8e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efd100097a0 tx=0x7efd1002b7a0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 -- 172.21.15.47:0/642819915 shutdown_connections 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 --2- 172.21.15.47:0/642819915 >> v2:172.21.15.52:3300/0 conn(0x7efd2013e800 0x7efd2013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 --2- 172.21.15.47:0/642819915 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd200a8e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 -- 172.21.15.47:0/642819915 >> 172.21.15.47:0/642819915 conn(0x7efd200b7280 msgr2=0x7efd200b7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:58.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.542+0000 7efd26d29640 1 -- 172.21.15.47:0/642819915 shutdown_connections 2024-03-30T07:16:58.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.543+0000 7efd26d29640 1 -- 172.21.15.47:0/642819915 wait complete. 2024-03-30T07:16:58.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.543+0000 7efd26d29640 10 monclient: build_initial_monmap 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.543+0000 7efd26d29640 10 monclient: monmap: 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.543+0000 7efd26d29640 1 librados: starting msgr at 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.543+0000 7efd26d29640 1 librados: starting objecter 2024-03-30T07:16:58.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 Processor -- start 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 -- start start 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 librados: setting wanted keys 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 librados: calling monclient init 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient: init 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient: _reopen_session rank -1 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd200719f0 0x7efd200b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-a con 0x7efd200719f0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd20131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-c con 0x7efd200b0c70 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): picked mon.noname-b con 0x7efd2013e800 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:58.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd201d9630 con 0x7efd200719f0 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd20141da0 con 0x7efd2013e800 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): start opening mon connection 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd20132800 con 0x7efd200b0c70 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): _renew_subs 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.544+0000 7efd26d29640 10 monclient(hunting): authenticate will time out at 1544.659962s 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd24ac5640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd200719f0 0x7efd200b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd17fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd20131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:42368/0 (socket says 172.21.15.47:42368) 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 1 -- 172.21.15.47:0/973512195 learned_addr learned my addr 172.21.15.47:0/973512195 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd24ac5640 10 monclient(hunting): get_auth_request con 0x7efd200719f0 auth_method 0 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd24ac5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:58.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd24ac5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd24ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 10 monclient(hunting): get_auth_request con 0x7efd2013e800 auth_method 0 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd252c6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd17fff640 10 monclient(hunting): get_auth_request con 0x7efd200b0c70 auth_method 0 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd17fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.545+0000 7efd17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd252c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd24ac5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd24ac5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:58.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd24ac5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.546+0000 7efd17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 -- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 msgr2=0x7efd20131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd20131290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 -- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd200719f0 msgr2=0x7efd200b61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd200719f0 0x7efd200b61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 10 monclient: _finish_hunting 0 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 monclient: found mon.noname-b 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.548+0000 7efd252c6640 1 -- 172.21.15.47:0/973512195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18009a70 con 0x7efd2013e800 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.550+0000 7efd252c6640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd1800efc0 tx=0x7efd1800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:58.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.550+0000 7efd157fa640 10 client.?.objecter ms_handle_connect 0x7efd2013e800 2024-03-30T07:16:58.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.551+0000 7efd157fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18009cd0 con 0x7efd2013e800 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: dump: 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.559 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: _finish_auth 0 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: _check_auth_tickets 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1800f040 con 0x7efd2013e800 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18003990 con 0x7efd2013e800 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 5 monclient: authenticate success, global_id 4420 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 10 monclient: _renew_subs 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:58.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10009400 con 0x7efd2013e800 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd157fa640 10 monclient: dump: 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd16ffd640 4 set_mon_vals no callback set 2024-03-30T07:16:58.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 10 monclient: _renew_subs 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd20144460 con 0x7efd2013e800 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.555+0000 7efd26d29640 1 librados: init done 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.556+0000 7efd26d29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.556+0000 7efd26d29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.556+0000 7efd26d29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:58.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.556+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd20144650 con 0x7efd2013e800 2024-03-30T07:16:58.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.575+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efd18014480 con 0x7efd2013e800 2024-03-30T07:16:58.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.576+0000 7efd157fa640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 0x7efd0c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:16:58.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.576+0000 7efd24ac5640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 0x7efd0c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:16:58.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.576+0000 7efd24ac5640 10 monclient: get_auth_request con 0x7efd0c06c590 auth_method 0 2024-03-30T07:16:58.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.576+0000 7efd157fa640 10 client.4420.objecter ms_handle_connect 0x7efd0c06c590 2024-03-30T07:16:58.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.576+0000 7efd24ac5640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 0x7efd0c06ea50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0011e0 tx=0x7efd1c007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:16:58.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.578+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd18059510 con 0x7efd2013e800 2024-03-30T07:16:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.578+0000 7efd157fa640 10 client.4420.objecter ms_dispatch 0x7efd20000b90 osd_map(27..27 src has 1..27) v4 2024-03-30T07:16:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.578+0000 7efd157fa640 3 client.4420.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:16:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.578+0000 7efd157fa640 3 client.4420.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:16:58.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.578+0000 7efd157fa640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-30T07:16:58.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.847+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7efd18055850 con 0x7efd2013e800 2024-03-30T07:16:58.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.847+0000 7efd157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:16:58.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:58.847+0000 7efd157fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:16:59.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.066+0000 7efd26d29640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-03-30T07:16:59.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.066+0000 7efd26d29640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-03-30T07:16:59.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.066+0000 7efd26d29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:16:59.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.066+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efd200b35b0 con 0x7efd2013e800 2024-03-30T07:16:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.074+0000 7efd157fa640 1 -- 172.21.15.47:0/973512195 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7efd18018080 con 0x7efd2013e800 2024-03-30T07:16:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.074+0000 7efd157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-03-30T07:16:59.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.074+0000 7efd157fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:16:59.075 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:16:59.075 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.075+0000 7efd26d29640 10 librados: watch_flush enter 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.075+0000 7efd26d29640 10 librados: watch_flush exit 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.075+0000 7efd26d29640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.075+0000 7efd26d29640 10 client.4420.objecter successfully canceled tick 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.075+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 msgr2=0x7efd0c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 0x7efd0c06ea50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0011e0 tx=0x7efd1c007d30 comp rx=0 tx=0).stop 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 10 monclient: shutdown 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:16:59.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 msgr2=0x7efd201317d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd1800efc0 tx=0x7efd1800c490 comp rx=0 tx=0).stop 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 shutdown_connections 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7efd0c06c590 0x7efd0c06ea50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7efd2013e800 0x7efd201317d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efd200b0c70 0x7efd20131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 --2- 172.21.15.47:0/973512195 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efd200719f0 0x7efd200b61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 >> 172.21.15.47:0/973512195 conn(0x7efd200b7280 msgr2=0x7efd20145270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 shutdown_connections 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 -- 172.21.15.47:0/973512195 wait complete. 2024-03-30T07:16:59.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:16:59.076+0000 7efd26d29640 1 librados: shutdown 2024-03-30T07:16:59.087 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-30T07:16:59.087 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T07:16:59.099 INFO:tasks.workunit:Pulling workunits from ref 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:16:59.099 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T07:16:59.100 DEBUG:teuthology.orchestra.run.smithi047:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T07:16:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T07:16:59.114 INFO:teuthology.orchestra.run.smithi047.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-30T07:16:59.115 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T07:16:59.169 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-30T07:16:59.169 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-30T07:16:59.223 INFO:tasks.workunit:timeout=6h 2024-03-30T07:16:59.223 INFO:tasks.workunit:cleanup=True 2024-03-30T07:16:59.223 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 2024-03-30T07:16:59.279 INFO:tasks.workunit.client.0.smithi047.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T07:17:01.697 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:29.508 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.698 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:29.510 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.741 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:29.553 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.745 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:29.556 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.749 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:29.527 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.751 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:29.529 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.752 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:29.564 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.794 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:29.606 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:01.797 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:29.575 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.707 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:49.518 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.791 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:49.603 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.815 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:01:49.627 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.827 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:49.605 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.848 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:49.626 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.864 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:01:49.642 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.864 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:49.676 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.869 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:49.681 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:21.896 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:01:49.708 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.779 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:09.590 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.842 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:09.653 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.862 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:09.674 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.875 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:09.686 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.878 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:09.657 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.902 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:09.680 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.918 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:09.696 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.924 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:09.736 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:41.945 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:09.756 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:17:48.521 INFO:tasks.workunit.client.0.smithi047.stderr:Updating files: 93% (11359/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:Note: switching to '55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83'. 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr: git switch -c 2024-03-30T07:17:49.247 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:Or undo this operation with: 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr: git switch - 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr: 2024-03-30T07:17:49.248 INFO:tasks.workunit.client.0.smithi047.stderr:HEAD is now at 55c4b44e755 Merge branch 'wip-65153-reef' of https://github.com/batrick/ceph into wip-yuri11-testing-2024-03-28-0753-reef 2024-03-30T07:17:49.252 DEBUG:teuthology.orchestra.run.smithi047:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T07:17:49.316 INFO:tasks.workunit.client.0.smithi047.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T07:17:49.318 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T07:17:49.318 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T07:17:49.964 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T07:17:50.025 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T07:17:50.068 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T07:17:50.069 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T07:17:50.069 INFO:tasks.workunit.client.0.smithi047.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T07:17:50.109 INFO:tasks.workunit.client.0.smithi047.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T07:17:50.112 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-03-30T07:17:50.113 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T07:17:50.169 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-30T07:17:50.170 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-30T07:17:50.170 DEBUG:teuthology.orchestra.run.smithi047:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-30T07:17:50.228 INFO:tasks.workunit.client.0.smithi047.stderr:+ parallel=1 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' '' = --serial ']' 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ color= 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -t 1 ']' 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ declare -A pids 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_aio 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_aio' 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_aio 2024-03-30T07:17:50.229 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_aio 2024-03-30T07:17:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40627 2024-03-30T07:17:50.230 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_aio on pid 40627' 2024-03-30T07:17:50.231 INFO:tasks.workunit.client.0.smithi047.stdout:test api_aio on pid 40627 2024-03-30T07:17:50.233 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40627 2024-03-30T07:17:50.233 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_aio_pp 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_aio_pp' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_aio_pp 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.234 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.235 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.236 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.236 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_aio_pp 2024-03-30T07:17:50.236 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40634 2024-03-30T07:17:50.236 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_aio_pp on pid 40634' 2024-03-30T07:17:50.236 INFO:tasks.workunit.client.0.smithi047.stdout:test api_aio_pp on pid 40634 2024-03-30T07:17:50.240 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40634 2024-03-30T07:17:50.240 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_io 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_io' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_io 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.241 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.242 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.243 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_io 2024-03-30T07:17:50.244 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40644 2024-03-30T07:17:50.245 INFO:tasks.workunit.client.0.smithi047.stdout:test api_io on pid 40644 2024-03-30T07:17:50.250 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_io on pid 40644' 2024-03-30T07:17:50.250 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40644 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_io_pp 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_io_pp' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_io_pp 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.251 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.252 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.253 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.254 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_io_pp 2024-03-30T07:17:50.255 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40655 2024-03-30T07:17:50.257 INFO:tasks.workunit.client.0.smithi047.stdout:test api_io_pp on pid 40655 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_io_pp on pid 40655' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40655 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_asio 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_asio' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_asio 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.264 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.265 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.266 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.267 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.268 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.269 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_asio 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40669 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_asio on pid 40669' 2024-03-30T07:17:50.270 INFO:tasks.workunit.client.0.smithi047.stdout:test api_asio on pid 40669 2024-03-30T07:17:50.278 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40669 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_list 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_list' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_list 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.279 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.280 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.281 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.282 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.283 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.284 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40627/exe 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_list 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.285 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.286 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40685 2024-03-30T07:17:50.286 INFO:tasks.workunit.client.0.smithi047.stdout:test api_list on pid 40685 2024-03-30T07:17:50.294 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_list on pid 40685' 2024-03-30T07:17:50.294 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40685 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_lock 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_lock' 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_lock 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.295 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.296 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.297 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.298 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_aio 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.299 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.300 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.301 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40634/exe 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_aio: /' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.302 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_lock 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40705 2024-03-30T07:17:50.303 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_lock on pid 40705' 2024-03-30T07:17:50.304 INFO:tasks.workunit.client.0.smithi047.stdout:test api_lock on pid 40705 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40705 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_lock_pp 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.316 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_lock_pp' 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.317 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_lock_pp 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.318 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.319 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.320 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_aio_pp 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.321 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.322 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.323 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40644/exe 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_lock_pp 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40726 2024-03-30T07:17:50.324 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_lock_pp on pid 40726' 2024-03-30T07:17:50.325 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-30T07:17:50.325 INFO:tasks.workunit.client.0.smithi047.stdout:test api_lock_pp on pid 40726 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40726 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_misc 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_misc' 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.336 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_misc 2024-03-30T07:17:50.337 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.338 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_io 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.339 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_io: /' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_io.log 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.340 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.341 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_misc 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.342 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40741 2024-03-30T07:17:50.343 INFO:tasks.workunit.client.0.smithi047.stdout:test api_misc on pid 40741 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_misc on pid 40741' 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40741 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.360 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.361 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40655/exe 2024-03-30T07:17:50.362 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_misc_pp 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_misc_pp' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.363 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.364 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.365 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.366 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_io_pp 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.367 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.368 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_misc_pp 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.369 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.370 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.371 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.372 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40669/exe 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.373 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.374 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.375 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_asio 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_misc_pp 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40780 2024-03-30T07:17:50.376 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_misc_pp on pid 40780' 2024-03-30T07:17:50.377 INFO:tasks.workunit.client.0.smithi047.stdout:test api_misc_pp on pid 40780 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40780 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_tier_pp 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.382 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_tier_pp' 2024-03-30T07:17:50.383 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.383 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.383 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.383 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_asio: /' 2024-03-30T07:17:50.384 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.385 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40705/exe 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_tier_pp 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.386 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40685/exe 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_tier_pp 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40791 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.387 INFO:tasks.workunit.client.0.smithi047.stdout:test api_tier_pp on pid 40791 2024-03-30T07:17:50.406 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_tier_pp on pid 40791' 2024-03-30T07:17:50.406 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.406 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40791 2024-03-30T07:17:50.406 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.406 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.407 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.408 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.409 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.410 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.411 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_list.log 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_lock 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_list 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.412 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_lock: /' 2024-03-30T07:17:50.413 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_pool 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_pool' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.414 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_list: /' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.415 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40726/exe 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40741/exe 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.416 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_pool 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.417 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.418 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_lock_pp 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_misc: /' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_misc 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.419 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_pool 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40849 2024-03-30T07:17:50.420 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_pool on pid 40849' 2024-03-30T07:17:50.421 INFO:tasks.workunit.client.0.smithi047.stdout:test api_pool on pid 40849 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40849 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_snapshots 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_snapshots' 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.428 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.429 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.430 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_snapshots 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.431 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.432 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.433 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_snapshots 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40874 2024-03-30T07:17:50.434 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_snapshots on pid 40874' 2024-03-30T07:17:50.435 INFO:tasks.workunit.client.0.smithi047.stdout:test api_snapshots on pid 40874 2024-03-30T07:17:50.451 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40874 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.452 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_snapshots_pp 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_snapshots_pp' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.453 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.454 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.455 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.456 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.456 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.456 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.456 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.456 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40780/exe 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_snapshots_pp 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.457 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.458 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.459 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.461 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.461 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_misc_pp 2024-03-30T07:17:50.461 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.461 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.462 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.463 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.464 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.465 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.466 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.467 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_snapshots_pp 2024-03-30T07:17:50.468 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40930 2024-03-30T07:17:50.469 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_snapshots_pp on pid 40930' 2024-03-30T07:17:50.469 INFO:tasks.workunit.client.0.smithi047.stdout:test api_snapshots_pp on pid 40930 2024-03-30T07:17:50.485 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40930 2024-03-30T07:17:50.485 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.485 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.485 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.485 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.486 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.487 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_stat 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.488 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.489 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_stat' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.490 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40791/exe 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.491 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.492 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.493 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.494 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.495 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_tier_pp 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.496 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.497 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_stat 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_stat 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40992 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-30T07:17:50.498 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_stat on pid 40992' 2024-03-30T07:17:50.499 INFO:tasks.workunit.client.0.smithi047.stdout:test api_stat on pid 40992 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=40992 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_stat_pp 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40849/exe 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40874/exe 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_stat_pp' 2024-03-30T07:17:50.506 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.507 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_stat_pp 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.508 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.509 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.510 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_snapshots 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_stat_pp 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41009 2024-03-30T07:17:50.511 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_stat_pp on pid 41009' 2024-03-30T07:17:50.512 INFO:tasks.workunit.client.0.smithi047.stdout:test api_stat_pp on pid 41009 2024-03-30T07:17:50.521 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_pool 2024-03-30T07:17:50.521 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41009 2024-03-30T07:17:50.521 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.521 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.521 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_pool: /' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.522 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_watch_notify 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_watch_notify' 2024-03-30T07:17:50.523 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.524 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.525 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.526 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40930/exe 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.527 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.528 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_watch_notify 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_watch_notify 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41038 2024-03-30T07:17:50.529 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_watch_notify on pid 41038' 2024-03-30T07:17:50.530 INFO:tasks.workunit.client.0.smithi047.stdout:test api_watch_notify on pid 41038 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41038 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.532 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.533 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_snapshots_pp 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [----------] Global test environment set-up. 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.363+0000 7f678b0d9780 10 monclient: get_monmap_and_config 2024-03-30T07:17:50.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.363+0000 7f678b0d9780 10 monclient: build_initial_monmap 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.363+0000 7f678b0d9780 10 monclient: monmap: 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 0 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 0.000000 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 0.000000 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.364+0000 7f678b0d9780 1 Processor -- start 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.364+0000 7f678b0d9780 1 -- start start 2024-03-30T07:17:50.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.364+0000 7f678b0d9780 10 monclient: init 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient: _reopen_session rank -1 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f4a2960 0x55ff0f4ab600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-a con 0x55ff0f4a2960 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f537770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-c con 0x55ff0f4a6840 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 0x55ff0f5d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-b con 0x55ff0f539590 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f4204c0 con 0x55ff0f4a2960 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f3bbb50 con 0x55ff0f539590 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f3bb7f0 con 0x55ff0f4a6840 2024-03-30T07:17:50.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): _renew_subs 2024-03-30T07:17:50.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.365+0000 7f678b0d9780 10 monclient(hunting): authenticate will time out at 1596.480350s 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.542 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_watch_notify_pp 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_watch_notify_pp' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.543 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.544 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.545 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.389+0000 7f67893d9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 0x55ff0f5d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f537770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f537770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:41144/0 (socket says 172.21.15.47:41144) 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 1 -- 172.21.15.47:0/4266562828 learned_addr learned my addr 172.21.15.47:0/4266562828 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 10 monclient(hunting): get_auth_request con 0x55ff0f4a6840 auth_method 0 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.392+0000 7f67883d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.397+0000 7f67893d9640 10 monclient(hunting): get_auth_request con 0x55ff0f539590 auth_method 0 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.397+0000 7f67893d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.397+0000 7f67893d9640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.397+0000 7f67893d9640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.400+0000 7f67893d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.400+0000 7f67893d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.400+0000 7f67893d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:17:50.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 10 monclient(hunting): handle_auth_done global_id 4450 payload 274 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 -- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 msgr2=0x55ff0f537770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f537770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 -- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f4a2960 msgr2=0x55ff0f4ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f4a2960 0x55ff0f4ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 10 monclient: _finish_hunting 0 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 monclient: found mon.noname-b 2024-03-30T07:17:50.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_watch_notify_pp 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.550 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_watch_notify_pp 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41065 2024-03-30T07:17:50.551 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_watch_notify_pp on pid 41065' 2024-03-30T07:17:50.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_atest api_watch_notify_pp on pid 41065 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41065 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.558 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_cmd 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_cmd' 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_cmd 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.559 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-30T07:17:50.560 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41009/exe 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.561 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_cmd 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41089 2024-03-30T07:17:50.562 INFO:tasks.workunit.client.0.smithi047.stdout:test api_cmd on pid 41089 2024-03-30T07:17:50.568 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_cmd on pid 41089' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41089 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_cmd_pp 2024-03-30T07:17:50.569 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_cmd_pp' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_cmd_pp 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.570 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.571 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_stat_pp 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.572 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_cmd_pp 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41115 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_cmd_pp on pid 41115' 2024-03-30T07:17:50.573 INFO:tasks.workunit.client.0.smithi047.stdout:test api_cmd_pp on pid 41115 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41115 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.575 INFO:tasks.workunit.client.0.smithi047.stdout:sio: 2024-03-30T07:17:50.404+0000 7f67893d9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.404+0000 7f67893d9640 1 -- 172.21.15.47:0/4266562828 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6778005d20 con 0x55ff0f539590 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.414+0000 7f67893d9640 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 0x55ff0f5d2a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55ff0f4a30d0 tx=0x7f677804c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef42ebd4680cc45 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 1 -- 172.21.15.47:0/4266562828 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677800a230 con 0x55ff0f539590 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: dump: 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 1 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 18 (reef) 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:17:50.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: _finish_auth 0 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: _check_auth_tickets 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 1 -- 172.21.15.47:0/4266562828 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677804acb0 con 0x55ff0f539590 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 1 -- 172.21.15.47:0/4266562828 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778052680 con 0x55ff0f539590 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f67873d5640 10 monclient: dump: 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 1 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 18 (reef) 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f678b0d9780 5 monclient: authenticate success, global_id 4450 2024-03-30T07:17:50.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f678b0d9780 10 monclient: get_monmap_and_config success 2024-03-30T07:17:50.585 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.585 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.585 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.586 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.587 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.588 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_service 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_service' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.589 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/40992/exe 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.590 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.591 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.422+0000 7f678b0d9780 4 set_mon_vals no callback set 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.423+0000 7f678b0d9780 10 monclient: shutdown 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.423+0000 7f678b0d9780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.423+0000 7f678b0d9780 1 -- 172.21.15.47:0/4266562828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 msgr2=0x55ff0f5d2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.423+0000 7f678b0d9780 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 0x55ff0f5d2a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55ff0f4a30d0 tx=0x7f677804c000 comp rx=0 tx=0).stop 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 -- 172.21.15.47:0/4266562828 shutdown_connections 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f539590 0x55ff0f5d2a90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f537770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 --2- 172.21.15.47:0/4266562828 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f4a2960 0x55ff0f4ab600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.592 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 -- 172.21.15.47:0/4266562828 >> 172.21.15.47:0/4266562828 conn(0x55ff0f4ac260 msgr2=0x55ff0f4a5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 -- 172.21.15.47:0/4266562828 shutdown_connections 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 -- 172.21.15.47:0/4266562828 wait complete. 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 10 monclient: build_initial_monmap 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 10 monclient: monmap: 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 0 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 0.000000 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 0.000000 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.593 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 librados: starting msgr at 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.425+0000 7f678b0d9780 1 librados: starting objecter 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.426+0000 7f678b0d9780 1 Processor -- start 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 -- start start 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 librados: setting wanted keys 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 librados: calling monclient init 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient: init 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient: _reopen_session rank -1 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:17:50.594 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f4a2960 0x55ff0f46bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.598 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.598 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.598 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.598 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_service 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.599 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.600 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.601 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.602 INFO:tasks.workunit.client.0.smithi047.stdout: test api_service on pid 41157 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_service 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41157 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_service on pid 41157' 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41157 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.613 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.614 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.615 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.616 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_stat 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_stat: /' 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_service_pp 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_service_pp' 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.617 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.618 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_service_pp 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.619 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41038/exe 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.620 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.621 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.622 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_watch_notify 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_service_pp 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41200 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_service_pp on pid 41200' 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.623 INFO:tasks.workunit.client.0.smithi047.stdout:test api_service_pp on pid 41200 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.625 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41200 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.626 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-b con 0x55ff0f4a2960 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f52a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-c con 0x55ff0f4a6840 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f539590 0x55ff0f52ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): picked mon.noname-a con 0x55ff0f539590 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f39e510 con 0x55ff0f539590 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f52b150 con 0x55ff0f4a2960 2024-03-30T07:17:50.627 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): start opening mon connection 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff0f52a200 con 0x55ff0f4a6840 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): _renew_subs 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.428+0000 7f678b0d9780 10 monclient(hunting): authenticate will time out at 1596.544115s 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f52a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f52a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:41178/0 (socket says 172.21.15.47:41178) 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 1 -- 172.21.15.47:0/106249272 learned_addr learned my addr 172.21.15.47:0/106249272 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 10 monclient(hunting): get_auth_request con 0x55ff0f4a6840 auth_method 0 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.439+0000 7f67883d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.440+0000 7f67883d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.440+0000 7f67883d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:17:50.628 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.440+0000 7f67883d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_c_write_operations 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-30T07:17:50.633 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_c_write_operations' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_c_write_operations 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.634 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.635 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_c_write_operations 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41212 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.636 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_c_write_operations on pid 41212' 2024-03-30T07:17:50.637 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.637 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.446+0000 7f67883d7640 10 monclient(hunting): handle_auth_done global_id 4349 paylotest api_c_write_operations on pid 41212 2024-03-30T07:17:50.644 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41212 2024-03-30T07:17:50.644 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.644 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_c_read_operations 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.645 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_c_read_operations' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.646 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41065/exe 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_c_read_operations 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.647 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.648 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.649 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.650 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_c_read_operations 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41229 2024-03-30T07:17:50.651 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_c_read_operations on pid 41229' 2024-03-30T07:17:50.652 INFO:tasks.workunit.client.0.smithi047.stdout:test api_c_read_operations on pid 41229 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41229 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.670 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.671 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.672 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s api_cls_remote_reads 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' api_cls_remote_reads' 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.673 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.674 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.675 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.676 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.677 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.678 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.679 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo api_cls_remote_reads 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41115/exe 2024-03-30T07:17:50.680 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.681 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.682 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=api_cls_remote_reads 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41263 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.683 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.684 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test api_cls_remote_reads on pid 41263' 2024-03-30T07:17:50.685 INFO:tasks.workunit.client.0.smithi047.stdout:test api_cls_remote_reads on pid 41263 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41263 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.697 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.698 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41157/exe 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_cmd_pp 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s list_parallel 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' list_parallel' 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.699 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.700 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_service 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.701 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.702 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_service.log 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_service: /' 2024-03-30T07:17:50.703 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo list_parallel 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.704 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.705 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41212/exe 2024-03-30T07:17:50.706 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=list_parallel 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41298 2024-03-30T07:17:50.707 INFO:tasks.workunit.client.0.smithi047.stdout:test list_parallel on pid 41298 2024-03-30T07:17:50.723 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test list_parallel on pid 41298' 2024-03-30T07:17:50.723 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41298 2024-03-30T07:17:50.723 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41089/exe 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.724 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s open_pools_parallel 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' open_pools_parallel' 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-30T07:17:50.725 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_c_write_operations 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_cmd: /' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.726 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41200/exe 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.727 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.728 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.729 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.730 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.731 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.732 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.733 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_service_pp 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo open_pools_parallel 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.734 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_cmd 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.735 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.736 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout:ad 824 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 -- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f4a2960 msgr2=0x55ff0f46bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x55ff0f4a2960 0x55ff0f46bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 -- 172.21.15.47:0/106249272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f539590 msgr2=0x55ff0f52ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x55ff0f539590 0x55ff0f52ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 10 monclient: _finish_hunting 0 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 monclient: found mon.noname-c 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.447+0000 7f67883d7640 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff0f3cac10 con 0x55ff0f4a6840 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.462+0000 7f67883d7640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f52a6d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f67800021f0 tx=0x7f678000d5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.462+0000 7f67863d3640 10 client.?.objecter ms_handle_connect 0x55ff0f4a6840 2024-03-30T07:17:50.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.462+0000 7f67863d3640 10 client.?.objecter resend_mon_ops 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780004910 con 0x55ff0f4a6840 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: dump: 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 1 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 18 (reef) 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:17:50.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.739 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: _finish_auth 0 2024-03-30T07:17:50.739 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: _check_auth_tickets 2024-03-30T07:17:50.739 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:17:50.739 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41229/exe 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=open_pools_parallel 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41359 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test open_pools_parallel on pid 41359' 2024-03-30T07:17:50.740 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780004a70 con 0x55ff0f4a684test open_pools_parallel on pid 41359 2024-03-30T07:17:50.745 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41359 2024-03-30T07:17:50.745 INFO:tasks.workunit.client.0.smithi047.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-30T07:17:50.745 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.746 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:++ printf %25s delete_pools_parallel 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+ r=' delete_pools_parallel' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.747 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.748 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_c_read_operations 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:++ awk '{print $1}' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:++ echo delete_pools_parallel 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ ff=delete_pools_parallel 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41389 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.749 INFO:tasks.workunit.client.0.smithi047.stderr:+ echo 'test delete_pools_parallel on pid 41389' 2024-03-30T07:17:50.750 INFO:tasks.workunit.client.0.smithi047.stdout:test delete_pools_parallel on pid 41389 2024-03-30T07:17:50.777 INFO:tasks.workunit.client.0.smithi047.stderr:+ pids[$f]=41389 2024-03-30T07:17:50.777 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.777 INFO:tasks.workunit.client.0.smithi047.stderr:+ ret=0 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' 1 -eq 1 ']' 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40791 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40791 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -f /etc/bashrc ']' 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:+ . /etc/bashrc 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -z '' ']' 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ BASHRCSOURCED=Y 2024-03-30T07:17:50.778 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ shopt -q login_shell 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.779 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.780 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.781 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.782 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41263/exe 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.783 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+++ umask 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.784 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.785 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41298/exe 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' 0022 -eq 0 ']' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ SHELL=/bin/bash 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-30T07:17:50.786 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.787 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_list_parallel 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ list_parallel: /' 2024-03-30T07:17:50.788 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.789 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'grep=grep --color=auto' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorls.sh 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' '!' -t 0 ']' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:+++ return 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-30T07:17:50.790 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export S_COLORS=auto 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ S_COLORS=auto 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.791 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.792 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.793 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/gawk.sh 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-30T07:17:50.794 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/lang.sh 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/locale 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41389/exe 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-30T07:17:50.795 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:++++ export LANG=en_US.UTF-8 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:++++ LANG=en_US.UTF-8 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_delete_pools_parallel 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.796 INFO:tasks.workunit.client.0.smithi047.stderr:+++ LANG=en_US.UTF-8 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ unset LANG_backup config 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n '' ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' dumb = linux ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/less.sh 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ for i in /etc/profile.d/*.sh 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ '[' '' ']' 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:++ . /etc/profile.d/which2.sh 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-30T07:17:50.797 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+++++ readlink /proc/41359/exe 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:++++ basename /usr/bin/bash 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_declare='declare -f' 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+++ which_opt=-f 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export which_declare 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+++ export -f which 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset i 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:++ unset -f pathmunge 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ export PATH 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ unset rc 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ ceph_test_rados_open_pools_parallel 2024-03-30T07:17:50.798 INFO:tasks.workunit.client.0.smithi047.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout:0 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780004e20 con 0x55ff0f4a6840 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67863d3640 10 monclient: dump: 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: epoch 1 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:17:50.799 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: min_mon_release 18 (reef) 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: election_strategy: 1 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f67873d5640 4 set_mon_vals no callback set 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 5 monclient: authenticate success, global_id 4349 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 monclient: _renew_subs 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ff0f3be050 con 0x55ff0f4a6840 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 monclient: _renew_subs 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:17:50.800 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff0f3bdab0 con 0x55ff0f4a6840 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 1 librados: init done 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.513+0000 7f678b0d9780 10 librados: wait_for_osdmap waiting 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.624+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6780006070 con 0x55ff0f4a6840 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.625+0000 7f67863d3640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f677406a800 0x7f677406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.625+0000 7f6788bd8640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f677406a800 0x7f677406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:17:50.801 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.625+0000 7f6788bd8640 10 monclient: get_auth_request con 0x7f677406a800 auth_method 0 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.625+0000 7f6788bd8640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f677406a800 0x7f677406ccc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f677c000f60 tx=0x7f677c0013f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.625+0000 7f67863d3640 10 client.4349.objecter ms_handle_connect 0x7f677406a800 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6780056760 con 0x55ff0f4a6840 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(27..27 src has 1..27) v4 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [27,27] > 0 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding full epoch 27 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f678b0d9780 10 librados: wait_for_osdmap done waiting 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f678b0d9780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:17:53.144 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.627+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55ff0f418f80 con 0x55ff0f4a6840 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6780052aa0 con 0x55ff0f4a6840 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f67863d3640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f6786bd4640 10 client.4349.objecter _get_latest_version latest 27, have it 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f678b0d9780 10 client.4349.objecter create_pool name=ceph_test_rados_api_asio 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f678b0d9780 10 client.4349.objecter pool_op_submit 1 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f678b0d9780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:50.685+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55ff0f429f00 con 0x55ff0f4a6840 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v29) v1 ==== 43+0+0 (secure 0 0 0) 0x7f6780018030 con 0x55ff0f4a6840 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 pool_op_reply(tid 1 (0) Success v29) v1 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 client.4349.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v29) v1 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 client.4349.objecter have request 1 at 0x55ff0f5382f0 Op: create 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 20 client.4349.objecter waiting for client to reach epoch 29 before calling back 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:17:53.145 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 monclient: _renew_subs 2024-03-30T07:17:53.146 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:17:53.146 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=28}) v3 -- 0x7f6774070540 con 0x55ff0f4a6840 2024-03-30T07:17:53.146 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.048+0000 7f67863d3640 10 client.4349.objecter done 2024-03-30T07:17:53.149 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== mon.2 v2:172.21.15.47:3301/0 8 ==== osd_map(28..29 src has 1..29) v4 ==== 11843+0+0 (secure 0 0 0) 0x7f678005a4e0 con 0x55ff0f4a6840 2024-03-30T07:17:53.149 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(28..29 src has 1..29) v4 2024-03-30T07:17:53.149 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [28,29] > 27 2024-03-30T07:17:53.149 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 28 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 29 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 10 librados: set snap read head -> 1 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 10 librados: writefull oid=exist nspace= 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f525b10 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 29 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x55ff0f46a870 0x55ff0f526510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [writefull 0~5 in=5b] tid 2 osd.6 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:17:53.150 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 2 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 15 client.4349.objecter _send_op 2 to 16.1 on osd.6 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:2 16.1 16:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) v8 -- 0x55ff0f526c50 con 0x55ff0f46a870 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.143+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.144+0000 7f67893d9640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x55ff0f46a870 0x55ff0f526510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.144+0000 7f67893d9640 10 monclient: get_auth_request con 0x55ff0f46a870 auth_method 0 2024-03-30T07:17:53.151 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.148+0000 7f67893d9640 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x55ff0f46a870 0x55ff0f526510 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.148+0000 7f67863d3640 10 client.4349.objecter ms_handle_connect 0x55ff0f46a870 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.428+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.428+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.428+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:53.429+0000 7f67853d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:54.430+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:54.430+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:54.430+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:55.430+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.606 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:55.430+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:55.430+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:55.514+0000 7f678ae3c640 10 client.4349.objecter tick 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:56.430+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:56.430+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:56.430+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:57.430+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:57.430+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:57.430+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:58.430+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:58.430+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:58.430+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:59.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:59.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:17:59.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.607 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.515+0000 7f678ae3c640 10 client.4349.objecter tick 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 1 ==== osd_map(30..31 src has 1..31) v4 ==== 8663+0+0 (crc 0 0 0) 0x7f677800a210 con 0x55ff0f46a870 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(30..31 src has 1..31) v4 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [30,31] > 29 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 30 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 10 client.4349.objecter checking op 2 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 30 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:00.608 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:02.174 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 31 2024-03-30T07:18:02.174 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 10 client.4349.objecter checking op 2 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 31 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:00.604+0000 7f67863d3640 20 client.4349.objecter 2 16.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:01.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:01.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:01.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 2 ==== osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(2 exist [writefull 0~5] v31'1 uv1 ondisk = 0) v8 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 2 ondisk uv 1 in 16.1 attempt 0 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:02.175 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 2 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 15 client.4349.objecter _finish_op 2 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 2 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 librados: Objecter returned from writefull r=0 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f525b10 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 31 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~256] tid 3 osd.6 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:02.176 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 3 2024-03-30T07:18:02.177 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 15 client.4349.objecter _send_op 3 to 16.1 on osd.6 2024-03-30T07:18:02.353 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:30.165 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.440 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:30.252 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.451 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:30.262 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.452 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:30.264 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.587 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:30.399 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.593 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:30.404 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.646 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:30.424 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.649 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:30.427 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:02.865 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:30.643 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:03.826 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:3 16.1 16:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55ff0f527000 con 0x55ff0f46a870 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f527440 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 31 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~256] tid 4 osd.6 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 4 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 15 client.4349.objecter _send_op 4 to 16.1 on osd.6 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:4 16.1 16:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55ff0f527930 con 0x55ff0f46a870 2024-03-30T07:18:03.827 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.173+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:02.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.431+0000 7f67853d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 3 ==== osd_map(32..33 src has 1..34) v4 ==== 1968+0+0 (crc 0 0 0) 0x7f6778053dd0 con 0x55ff0f46a870 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(32..33 src has 1..34) v4 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [32,33] > 31 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 32 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 10 client.4349.objecter checking op 3 2024-03-30T07:18:03.828 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 32 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.829 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:03.884 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 10 client.4349.objecter checking op 4 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 32 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 33 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 10 client.4349.objecter checking op 3 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 33 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 10 client.4349.objecter checking op 4 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 33 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter 3 16.d3a59201 osd.6 exist [read 0~256] 2024-03-30T07:18:03.885 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.825+0000 7f67863d3640 20 client.4349.objecter 4 16.a7bbaa1 osd.6 noexist [read 0~256] 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 3 ondisk uv 1 in 16.1 attempt 0 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 3 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 15 client.4349.objecter _finish_op 3 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 3 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.837+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:03.886 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 4 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1710 ms) 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 4 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 15 client.4349.objecter _finish_op 4 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 4 2024-03-30T07:18:03.887 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f525b10 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 33 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~256] tid 5 osd.6 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 5 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 15 client.4349.objecter _send_op 5 to 16.1 on osd.6 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:5 16.1 16:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55ff0f527440 con 0x55ff0f46a870 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f4697b0 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 33 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~256] tid 6 osd.6 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 6 2024-03-30T07:18:03.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 15 client.4349.objecter _send_op 6 to 16.1 on osd.6 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:6 16.1 16:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55ff0f469ca0 con 0x55ff0f46a870 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:03.883+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 5 ondisk uv 1 in 16.1 attempt 0 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 5 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter _finish_op 5 2024-03-30T07:18:04.344 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 5 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 6 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 6 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter _finish_op 6 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 6 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.342+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:04.345 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (459 ms) 2024-03-30T07:18:04.346 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-30T07:18:04.346 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f4697b0 2024-03-30T07:18:04.346 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 33 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:04.346 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:04.815 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:04.815 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:04.815 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~256] tid 7 osd.6 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 7 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 15 client.4349.objecter _send_op 7 to 16.1 on osd.6 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:7 16.1 16:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55ff0f525b10 con 0x55ff0f46a870 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f62ce00 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 33 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~256] tid 8 osd.6 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:04.816 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 8 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 15 client.4349.objecter _send_op 8 to 16.1 on osd.6 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:8 16.1 16:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) v8 -- 0x55ff0f62d2f0 con 0x55ff0f46a870 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.343+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 8 ==== osd_map(34..34 src has 1..35) v4 ==== 1007+0+0 (crc 0 0 0) 0x7f677806b040 con 0x55ff0f46a870 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(34..34 src has 1..35) v4 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [34,34] > 33 2024-03-30T07:18:04.817 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 34 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 10 client.4349.objecter checking op 7 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 34 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 10 client.4349.objecter checking op 8 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 34 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter 7 16.d3a59201 osd.6 exist [read 0~256] 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67863d3640 20 client.4349.objecter 8 16.a7bbaa1 osd.6 noexist [read 0~256] 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 7 ondisk uv 1 in 16.1 attempt 0 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 7 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter _finish_op 7 2024-03-30T07:18:04.818 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 7 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (472 ms) 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 8 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 8 2024-03-30T07:18:04.819 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter _finish_op 8 2024-03-30T07:18:05.479 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:05.479 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 8 2024-03-30T07:18:05.479 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:05.479 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f4697b0 write_seq 1 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f46a0d0 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 34 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:05.480 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [write 0~5 in=5b] tid 9 osd.6 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 9 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 15 client.4349.objecter _send_op 9 to 16.1 on osd.6 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:9 16.1 16:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34) v8 -- 0x55ff0f4698c0 con 0x55ff0f46a870 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:05.481 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:04.814+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 11 ==== osd_map(35..35 src has 1..35) v4 ==== 992+0+0 (crc 0 0 0) 0x7f6778053a00 con 0x55ff0f46a870 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(35..35 src has 1..35) v4 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [35,35] > 34 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 35 2024-03-30T07:18:05.482 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 10 client.4349.objecter checking op 9 2024-03-30T07:18:05.483 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 35 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:05.483 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:05.483 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.477+0000 7f67863d3640 20 client.4349.objecter 9 16.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:05.516+0000 7f678ae3c640 10 client.4349.objecter tick 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:06.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:06.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:06.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:07.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:07.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:07.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:08.865 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 12 ==== osd_op_reply(9 exist [write 0~5] v35'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(9 exist [write 0~5] v35'2 uv2 ondisk = 0) v8 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 9 ondisk uv 2 in 16.1 attempt 0 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 9 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 15 client.4349.objecter _finish_op 9 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 9 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f4697b0 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (4049 ms) 2024-03-30T07:18:08.866 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f46a5c0 write_seq 2 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f46a0d0 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 35 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [write 0~5 in=5b] tid 10 osd.6 2024-03-30T07:18:08.867 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:09.432 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 10 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 15 client.4349.objecter _send_op 10 to 16.1 on osd.6 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:10 16.1 16:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35) v8 -- 0x55ff0f4698c0 con 0x55ff0f46a870 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.863+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 13 ==== osd_map(36..39 src has 1..40) v4 ==== 1819+0+0 (crc 0 0 0) 0x7f6778053bc0 con 0x55ff0f46a870 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(36..39 src has 1..40) v4 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [36,39] > 35 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 36 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 10 client.4349.objecter checking op 10 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 36 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:09.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 37 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 10 client.4349.objecter checking op 10 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 37 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 38 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 10 client.4349.objecter checking op 10 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 38 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 39 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 10 client.4349.objecter checking op 10 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 39 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:09.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:09.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:08.924+0000 7f67863d3640 20 client.4349.objecter 10 16.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-30T07:18:09.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:09.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:09.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:09.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:09.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 14 ==== osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) v8 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 10 ondisk uv 3 in 16.1 attempt 0 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 10 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 15 client.4349.objecter _finish_op 10 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:10.433 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 10 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f46a5c0 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1557 ms) 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f469aa0 write_seq 3 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f46a0d0 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 39 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [write 0~5 in=5b] tid 11 osd.6 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:10.434 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 11 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 15 client.4349.objecter _send_op 11 to 16.1 on osd.6 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:11 16.1 16:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v8 -- 0x55ff0f6105c0 con 0x55ff0f46a870 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.419+0000 7f678b0d9780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-30T07:18:10.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 15 ==== osd_map(40..41 src has 1..41) v4 ==== 1283+0+0 (crc 0 0 0) 0x7f67780535c0 con 0x55ff0f46a870 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(40..41 src has 1..41) v4 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [40,41] > 39 2024-03-30T07:18:11.088 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 40 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 10 client.4349.objecter checking op 11 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 40 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 41 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 10 client.4349.objecter checking op 11 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.443+0000 7f67863d3640 20 client.4349.objecter 11 16.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:10.516+0000 7f678ae3c640 10 client.4349.objecter tick 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 16 ==== osd_op_reply(11 exist [write 0~5] v41'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(11 exist [write 0~5] v41'4 uv4 ondisk = 0) v8 2024-03-30T07:18:11.089 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 11 ondisk uv 4 in 16.1 attempt 0 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 11 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 15 client.4349.objecter _finish_op 11 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 11 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.085+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f469aa0 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (666 ms) 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-30T07:18:11.090 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ecde0 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~0] tid 12 osd.6 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 12 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 15 client.4349.objecter _send_op 12 to 16.1 on osd.6 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:12 16.1 16:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5ed2d0 con 0x55ff0f46a870 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.126 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~0] tid 13 osd.6 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 13 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 15 client.4349.objecter _send_op 13 to 16.1 on osd.6 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:13 16.1 16:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5eff40 con 0x55ff0f46a870 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.086+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 17 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-30T07:18:11.127 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 12 ondisk uv 4 in 16.1 attempt 0 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 12 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter _finish_op 12 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 12 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 18 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 13 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:11.128 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (38 ms) 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 13 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter _finish_op 13 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 13 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.123+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~0] tid 14 osd.6 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 14 2024-03-30T07:18:11.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 15 client.4349.objecter _send_op 14 to 16.1 on osd.6 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:14 16.1 16:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5ecde0 con 0x55ff0f46a870 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5eff40 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.212 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~0] tid 15 osd.6 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 15 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 15 client.4349.objecter _send_op 15 to 16.1 on osd.6 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:15 16.1 16:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5f0430 con 0x55ff0f46a870 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.124+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (86 ms) 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 19 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 14 ondisk uv 4 in 16.1 attempt 0 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 14 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter _finish_op 14 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 14 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 20 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 15 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 15 2024-03-30T07:18:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter _finish_op 15 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 15 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [read 0~0] tid 16 osd.6 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 16 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 15 client.4349.objecter _send_op 16 to 16.1 on osd.6 2024-03-30T07:18:11.215 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:16 16.1 16:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5ecde0 con 0x55ff0f46a870 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f630a90 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base noexist @16 precalc_pgid 0 pgid 0.0 is_read 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _calc_target target noexist @16 -> pgid 16.a7bbaa1 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.a7bbaa1 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid noexist '@16' '@16' [read 0~0] tid 17 osd.6 2024-03-30T07:18:11.216 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.466 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 17 2024-03-30T07:18:11.466 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 15 client.4349.objecter _send_op 17 to 16.1 on osd.6 2024-03-30T07:18:11.466 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:17 16.1 16:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f6305d0 con 0x55ff0f46a870 2024-03-30T07:18:11.466 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.466 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.209+0000 7f678b0d9780 5 client.4349.objecter 2 in flight 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 21 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 16 ondisk uv 4 in 16.1 attempt 0 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 5 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 16 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter _finish_op 16 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 5 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 16 2024-03-30T07:18:11.467 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 5 client.4349.objecter 1 in flight 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 22 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 17 ondisk uv 0 in 16.1 attempt 0 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 17 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter _finish_op 17 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 17 2024-03-30T07:18:11.468 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:13.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (255 ms) 2024-03-30T07:18:13.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-30T07:18:13.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f5ecde0 write_seq 4 2024-03-30T07:18:13.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [writefull 0~5 in=5b] tid 18 osd.6 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 18 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 15 client.4349.objecter _send_op 18 to 16.1 on osd.6 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:18 16.1 16:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5f0f50 con 0x55ff0f46a870 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:11.464+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:12.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:12.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:13.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:12.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.431+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.431+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.431+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.431+0000 7f67853d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 23 ==== osd_op_reply(18 exist [writefull 0~5] v41'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(18 exist [writefull 0~5] v41'5 uv5 ondisk = 0) v8 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 18 ondisk uv 5 in 16.1 attempt 0 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 18 2024-03-30T07:18:13.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 15 client.4349.objecter _finish_op 18 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 18 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f5ecde0 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2111 ms) 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f5f1300 write_seq 5 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:14.269 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 41 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [writefull 0~5 in=5b] tid 19 osd.6 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 19 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 15 client.4349.objecter _send_op 19 to 16.1 on osd.6 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:19 16.1 16:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) v8 -- 0x55ff0f5f0f50 con 0x55ff0f46a870 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:13.575+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 24 ==== osd_map(42..43 src has 1..43) v4 ==== 951+0+0 (crc 0 0 0) 0x7f67780535c0 con 0x55ff0f46a870 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_map(42..43 src has 1..43) v4 2024-03-30T07:18:14.270 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 3 client.4349.objecter handle_osd_map got epochs [42,43] > 41 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 42 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 10 client.4349.objecter checking op 19 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 42 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 3 client.4349.objecter handle_osd_map decoding incremental epoch 43 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 10 client.4349.objecter checking op 19 2024-03-30T07:18:14.271 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter _calc_target epoch 43 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:14.888 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.267+0000 7f67863d3640 20 client.4349.objecter 19 16.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.432+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.432+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.432+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.885+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 25 ==== osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(19 exist [writefull 0~5] v43'6 uv6 ondisk = 0) v8 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 19 ondisk uv 6 in 16.1 attempt 0 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 19 2024-03-30T07:18:14.889 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 15 client.4349.objecter _finish_op 19 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 19 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f5f1300 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1310 ms) 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 librados: queue_aio_write 0x55ff0f3bb5e0 completion 0x55ff0f5ecef0 write_seq 6 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 10 client.4349.objecter _op_submit op 0x55ff0f5ed790 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 client.4349.objecter _calc_target epoch 43 base exist @16 precalc_pgid 0 pgid 0.0 is_write 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 client.4349.objecter _calc_target target exist @16 -> pgid 16.d3a59201 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 10 client.4349.objecter _calc_target raw pgid 16.d3a59201 -> actual 16.1 acting [6,1] primary 6 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 client.4349.objecter _get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 10 client.4349.objecter _op_submit oid exist '@16' '@16' [writefull 0~5 in=5b] tid 20 osd.6 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 client.4349.objecter get_session s=0x55ff0f536d40 osd=6 3 2024-03-30T07:18:14.890 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 15 client.4349.objecter _session_op_assign 6 20 2024-03-30T07:18:14.891 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 15 client.4349.objecter _send_op 20 to 16.1 on osd.6 2024-03-30T07:18:14.891 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- osd_op(unknown.0.0:20 16.1 16:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) v8 -- 0x55ff0f6115d0 con 0x55ff0f46a870 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:14.886+0000 7f678b0d9780 5 client.4349.objecter 1 in flight 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.432+0000 7f67853d1640 10 monclient: tick 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.432+0000 7f67853d1640 10 monclient: _check_auth_tickets 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.432+0000 7f67853d1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 <== osd.6 v2:172.21.15.52:6802/35315 26 ==== osd_op_reply(20 exist [writefull 0~5] v43'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f6778062020 con 0x55ff0f46a870 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 10 client.4349.objecter ms_dispatch 0x55ff0f410060 osd_op_reply(20 exist [writefull 0~5] v43'7 uv7 ondisk = 0) v8 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 10 client.4349.objecter in handle_osd_op_reply 2024-03-30T07:18:15.511 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 7 client.4349.objecter handle_osd_op_reply 20 ondisk uv 7 in 16.1 attempt 0 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 10 client.4349.objecter op 0 rval 0 len 0 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 15 client.4349.objecter handle_osd_op_reply completed tid 20 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 15 client.4349.objecter _finish_op 20 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 4 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 15 client.4349.objecter _session_op_remove 6 20 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 5 client.4349.objecter 0 in flight 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.508+0000 7f67893d9640 20 librados: complete_aio_write 0x55ff0f5ecef0 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (623 ms) 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 10 librados: watch_flush enter 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 10 librados: watch_flush exit 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 10 client.4349.objecter close_session for osd.6 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x55ff0f46a870 msgr2=0x55ff0f526510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x55ff0f46a870 0x55ff0f526510 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 20 client.4349.objecter put_session s=0x55ff0f536d40 osd=6 1 2024-03-30T07:18:15.512 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-30T07:18:15.513 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 10 client.4349.objecter successfully canceled tick 2024-03-30T07:18:15.513 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f677406a800 msgr2=0x7f677406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:18:15.513 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f677406a800 0x7f677406ccc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f677c000f60 tx=0x7f677c0013f0 comp rx=0 tx=0).stop 2024-03-30T07:18:15.513 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 10 monclient: shutdown 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 msgr2=0x55ff0f52a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 --2- 172.21.15.47:0/106249272 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x55ff0f4a6840 0x55ff0f52a6d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f67800021f0 tx=0x7f678000d5f0 comp rx=0 tx=0).stop 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f67893d9640 1 -- 172.21.15.47:0/106249272 reap_dead start 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 shutdown_connections 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 >> 172.21.15.47:0/106249272 conn(0x55ff0f4ac260 msgr2=0x55ff0f4a57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 shutdown_connections 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 -- 172.21.15.47:0/106249272 wait complete. 2024-03-30T07:18:15.515 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.509+0000 7f678b0d9780 1 librados: shutdown 2024-03-30T07:18:15.516 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [----------] 12 tests from AsioRados (13336 ms total) 2024-03-30T07:18:15.516 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: 2024-03-30T07:18:15.516 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [----------] Global test environment tear-down 2024-03-30T07:18:15.516 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (25147 ms total) 2024-03-30T07:18:15.516 INFO:tasks.workunit.client.0.smithi047.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-30T07:18:19.893 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-30T07:18:19.893 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [----------] Global test environment set-up. 2024-03-30T07:18:19.893 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5359 ms) 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ OK ] LibRadosService.RegisterLate (60 ms) 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: cluster: 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: id: bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: health: HEALTH_WARN 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 18 pool(s) do not have an application enabled 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: services: 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-30T07:18:19.894 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 2m) 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: data: 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: pools: 26 pools, 777 pgs 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: objects: 2 objects, 449 KiB 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: pgs: 84.556% pgs unknown 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 14.286% pgs not active 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 657 unknown 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 111 creating+peering 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 9 active+clean 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: cluster: 2024-03-30T07:18:19.895 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: id: bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: health: HEALTH_WARN 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 7 pool(s) do not have an application enabled 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: services: 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 2m) 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: data: 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: pools: 29 pools, 849 pgs 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: objects: 3 objects, 449 KiB 2024-03-30T07:18:19.896 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: pgs: 45.112% pgs unknown 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 51.472% pgs not active 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 383 unknown 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 358 creating+peering 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 79 creating+activating 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 29 active+clean 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: io: 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: client: 200 B/s wr, 0 op/s rd, 0 op/s wr 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3480 ms) 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ OK ] LibRadosService.Status (20458 ms) 2024-03-30T07:18:19.897 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [----------] 4 tests from LibRadosService (29358 ms total) 2024-03-30T07:18:19.898 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: 2024-03-30T07:18:19.898 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [----------] Global test environment tear-down 2024-03-30T07:18:19.898 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29358 ms total) 2024-03-30T07:18:19.898 INFO:tasks.workunit.client.0.smithi047.stdout: api_service: [ PASSED ] 4 tests. 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5391 ms) 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (158 ms) 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20227 ms) 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-30T07:18:21.588 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: attempt 0 of 20 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: failed to deregister: 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "epoch": 14, 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "modified": "2024-03-30T07:18:17.953961+0000", 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "services": { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "laundry": { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "daemons": { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "summary": "", 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "close-test-pid41335": { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_epoch": 14, 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_stamp": "2024-03-30T07:18:16.396833+0000", 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "gid": 4732, 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "addr": "172.21.15.47:0/2111610674", 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "metadata": { 2024-03-30T07:18:21.589 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_release": "reef", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1055-g55c4b44e (55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83) reef (stable)", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version_short": "18.2.2-1055-g55c4b44e", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro": "centos", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_version": "9", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "foo": "bar", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "hostname": "smithi047", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_total_kb": "32494652", 2024-03-30T07:18:21.590 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "os": "Linux", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "this": "that" 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: }, 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "task_status": {} 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: }, 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "pid41280": { 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_epoch": 13, 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_stamp": "2024-03-30T07:17:59.914716+0000", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "gid": 4687, 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "addr": "172.21.15.47:0/1958068229", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "metadata": { 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_release": "reef", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1055-g55c4b44e (55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83) reef (stable)", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version_short": "18.2.2-1055-g55c4b44e", 2024-03-30T07:18:21.591 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-30T07:18:22.864 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:50.675 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:22.864 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:50.675 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:22.878 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:50.690 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:22.912 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:50.723 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:22.920 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:50.698 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:22.928 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:50.706 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:23.070 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:02:50.882 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:23.092 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:02:50.870 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:23.129 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:02:50.941 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:23.991 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:23.971+0000 12a60640 -1 osd.4 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4458.0:2 22.1a 22:5c6776db:::0.obj:head [write 0~115 in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:24.804 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] Global test environment set-up. 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:17:50.588+0000 7f33a4ef7780 -1 failed for service _ceph-mon._tcp 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:17:50.588+0000 7f33a4ef7780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (261 ms) 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-30T07:18:24.805 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2272 ms total) 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb5ea470 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbec21dc10 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb8c0c80 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb9636d0 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb964120 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb964b90 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb965640 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb966190 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb966d50 2024-03-30T07:18:24.806 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb9678b0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb966a80 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb776220 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb967570 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb776de0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb7778f0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb775fc0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb778880 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb777630 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb779410 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb779ba0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb740ba0 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb778630 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 0x55cbeb741b00 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: started 23 aios 2024-03-30T07:18:24.807 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb5ea470 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbec21dc10 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb8c0c80 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb9636d0 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb964120 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb964b90 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb965640 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb966190 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb966d50 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb9678b0 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb966a80 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb776220 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb967570 2024-03-30T07:18:24.808 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb776de0 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb7778f0 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb775fc0 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb778880 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb777630 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb779410 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb779ba0 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb740ba0 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb778630 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: waiting 0x55cbeb741b00 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: done. 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (32204 ms) 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (32204 ms total) 2024-03-30T07:18:24.809 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:18:25.211 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:25.202+0000 12a60640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:25.985 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:18:25.934+0000 ea58640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:2 9.1d 9:bc52eff6:test-rados-api-smithi047-41151-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:26.317 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:26.244+0000 12a60640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:27.384 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:18:27.292+0000 ea58640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:2 9.1d 9:bc52eff6:test-rados-api-smithi047-41151-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:27.399 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:27.318+0000 12a60640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4422.0:2 13.5 13:a256ef1e:test-rados-api-smithi047-41034-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:27.400 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:27.274+0000 12a60640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:27.403 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:27.370+0000 12a60640 -1 osd.7 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4549.0:2 12.8 12:128285fa:test-rados-api-smithi047-41196-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:28.381 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:28.372+0000 12a60640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4422.0:2 13.5 13:a256ef1e:test-rados-api-smithi047-41034-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:28.471 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:28.424+0000 12a60640 -1 osd.7 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4549.0:2 12.8 12:128285fa:test-rados-api-smithi047-41196-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:28.718 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:28.626+0000 12a60640 -1 osd.4 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:29.506 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:29.500+0000 12a60640 -1 osd.7 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4549.0:2 12.8 12:128285fa:test-rados-api-smithi047-41196-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:29.702 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:29.691+0000 12a60640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:30.042 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:29.964+0000 12a60640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:31.327 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:31.166+0000 12a60640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:31.332 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:18:31.115+0000 12a60640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 6.14 6:2ce23629:test-rados-api-smithi047-40753-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-03-30T07:18:32.020 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro": "centos", 2024-03-30T07:18:32.020 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-30T07:18:32.020 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_version": "9", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "foo": "bar", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "hostname": "smithi047", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_total_kb": "32494652", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "os": "Linux", 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "this": "that" 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: }, 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "task_status": {} 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.021 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: attempt 1 of 20 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: failed to deregister: 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: { 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "epoch": 16, 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "modified": "2024-03-30T07:18:23.957855+0000", 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "services": { 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "laundry": { 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "daemons": { 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "summary": "", 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "close-test-pid41335": { 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_epoch": 16, 2024-03-30T07:18:32.022 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "start_stamp": "2024-03-30T07:18:21.992912+0000", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "gid": 4741, 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "addr": "172.21.15.47:0/3017349180", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "metadata": { 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "arch": "x86_64", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_release": "reef", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1055-g55c4b44e (55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83) reef (stable)", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "ceph_version_short": "18.2.2-1055-g55c4b44e", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro": "centos", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "distro_version": "9", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "foo": "bar", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "hostname": "smithi047", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Mar 26 19:20:47 UTC 2024", 2024-03-30T07:18:32.023 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "kernel_version": "5.14.0-434.el9.x86_64", 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "mem_total_kb": "32494652", 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "os": "Linux", 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "this": "that" 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: }, 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: "task_status": {} 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: } 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: attempt 2 of 20 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (15695 ms) 2024-03-30T07:18:32.024 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (41471 ms total) 2024-03-30T07:18:32.025 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: 2024-03-30T07:18:32.025 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-30T07:18:32.025 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (41471 ms total) 2024-03-30T07:18:32.025 INFO:tasks.workunit.client.0.smithi047.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-30T07:18:32.971 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:32.964+0000 12a60640 -1 osd.2 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:34.320 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:34.302+0000 12a60640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:34.598 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:34.489+0000 12a60640 -1 osd.7 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:35.547 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:35.541+0000 12a60640 -1 osd.7 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:35.686 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:35.586+0000 12a60640 -1 osd.2 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:36.576 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:36.574+0000 12a60640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:36.813 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:36.780+0000 12a60640 -1 osd.7 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:36.996 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: Running main() from gmock_main.cc 2024-03-30T07:18:36.996 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-30T07:18:36.996 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [----------] Global test environment set-up. 2024-03-30T07:18:36.996 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-30T07:18:36.996 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4001 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4010 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3829 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3765 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3751 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6026 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5624 ms) 2024-03-30T07:18:36.997 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6250 ms) 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9349 ms) 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [----------] 9 tests from LibRadosPools (46605 ms total) 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [----------] Global test environment tear-down 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (46605 ms total) 2024-03-30T07:18:36.998 INFO:tasks.workunit.client.0.smithi047.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-30T07:18:37.824 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:37.773+0000 12a60640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:38.817 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:38.812+0000 12a60640 -1 osd.2 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:38.937 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:38.931+0000 12a60640 -1 osd.7 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:39.754 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:39.738+0000 12a60640 -1 osd.0 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:39.932 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:39.856+0000 12a60640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:39.956 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:39.951+0000 12a60640 -1 osd.7 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:40.935 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:40.906+0000 12a60640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:41.217 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:41.177+0000 12a60640 -1 osd.7 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:41.437 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:41.354+0000 12a60640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 26.14 26.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e31) 2024-03-30T07:18:42.051 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:42.023+0000 12a60640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:42.307 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:42.211+0000 12a60640 -1 osd.7 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:43.118 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:43.116+0000 12a60640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:43.218 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:11.029 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.255 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:11.066 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.280 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:11.058 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.288 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:11.066 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.315 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:11.127 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.326 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:43.324+0000 12a60640 -1 osd.7 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:43.329 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:11.140 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.370 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:11.181 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.392 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:11.170 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:43.453 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:11.264 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:18:44.175 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:44.173+0000 12a60640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:44.501 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:18:44.432+0000 12a60640 -1 osd.7 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:3 11.c 11:3786f598:test-rados-api-smithi047-40904-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-03-30T07:18:45.382 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:45.296+0000 12a60640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:46.545 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:46.388+0000 12a60640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:47.696 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:47.496+0000 12a60640 -1 osd.0 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:48.886 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:48.767+0000 12a60640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:49.980 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:49.974+0000 12a60640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:51.018 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:51.003+0000 12a60640 -1 osd.0 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:52.568 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:18:52.512+0000 12a60640 -1 osd.0 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30) 2024-03-30T07:18:53.454 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:53.387+0000 12a60640 -1 osd.2 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4720.0:3 40.6 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-03-30T07:18:54.608 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:54.598+0000 12a60640 -1 osd.2 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4720.0:3 40.6 40.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-03-30T07:18:56.210 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:18:56.149+0000 12a60640 -1 osd.2 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4720.0:3 40.6 40:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52) 2024-03-30T07:19:01.868 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:19:01.865+0000 12a60640 -1 osd.4 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:3 6.9 6:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:03.142 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:19:03.138+0000 12a60640 -1 osd.4 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:3 6.9 6:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:03.318 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:31.096 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.319 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:31.097 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.324 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:31.136 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.330 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:31.141 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.408 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:31.220 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.416 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:31.227 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.417 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:03:31.195 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.461 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:03:31.273 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:03.536 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:03:31.348 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:04.470 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:19:04.356+0000 12a60640 -1 osd.4 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:3 6.9 6:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:05.549 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:19:05.540+0000 12a60640 -1 osd.4 91 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:3 6.9 6:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:08.632 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:19:08.630+0000 12a60640 -1 osd.4 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:3 6.9 6:9294e480:nspace::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:11.202 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:19:11.160+0000 ea58640 -1 osd.1 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4422.0:8 13.1 13:81685698:test-rados-api-smithi047-41034-4::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68) 2024-03-30T07:19:12.353 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:19:12.316+0000 ea58640 -1 osd.1 96 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:5 24.2 24:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:16.591 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:19:16.589+0000 ea58640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:5 24.2 24:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:17.620 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:19:17.614+0000 ea58640 -1 osd.1 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4409.0:5 24.2 24:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=237b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (653 ms) 2024-03-30T07:19:25.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (164 ms) 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (94260 ms) 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (95077 ms total) 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (95077 ms total) 2024-03-30T07:19:25.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (100062 ms) 2024-03-30T07:19:30.649 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (100062 ms total) 2024-03-30T07:19:30.650 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: 2024-03-30T07:19:30.650 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-30T07:19:30.650 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (100062 ms total) 2024-03-30T07:19:30.650 INFO:tasks.workunit.client.0.smithi047.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-30T07:19:35.736 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-30T07:19:35.736 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (878 ms) 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (150 ms) 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (98525 ms) 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.133824+0000 mon.b [INF] from='client.? 172.21.15.47:0/3381790431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.164625+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 14312.831ms) (OSD_SLOW_PING_TIME_BACK) 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.170052+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 13524.449ms) (OSD_SLOW_PING_TIME_FRONT) 2024-03-30T07:19:35.737 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.170739+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.708800+0000 mon.a [WRN] Health check update: 0 slow ops, oldest one blocked for 49 sec, daemons [osd.1,osd.2] have slow ops. (SLOW_OPS) 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:23.964713+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:24.064951+0000 mon.b [INF] from='client.? 172.21.15.47:0/2416272721' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:24.441058+0000 mon.a [INF] from='client.? 172.21.15.47:0/1495872732' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi047-40714-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:24.445688+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi047-41102-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:24.522309+0000 mon.b [INF] from='client.? 172.21.15.47:0/3562570157' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi047-41225-6", "pg_num": 4}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:26.067458+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:26.477829+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi047-41225-6", "pg_num": 4}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:25.552085+0000 mon.b [INF] from='client.? 172.21.15.47:0/3898201513' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi047-40956-4"}]: dispatch 2024-03-30T07:19:35.738 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:25.711695+0000 mon.b [INF] from='client.? 172.21.15.47:0/3381790431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:37.958 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:25.752035+0000 mon.b [INF] from='client.? 172.21.15.47:0/156188911' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRados api_c_read_operations: Running main() from gmock_main.cc 2024-03-30T07:19:37.958 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-30T07:19:37.958 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-30T07:19:37.958 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-30T07:19:37.958 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (14539 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (51740 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (4065 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (6357 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (269 ms) 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-30T07:19:37.959 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (972 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (410 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (1344 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (72 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (1651 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (874 ms) 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-30T07:19:37.960 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (115 ms) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (1234 ms) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (8211 ms) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (283 ms) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (858 ms) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (92994 ms total) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (107409 ms total) 2024-03-30T07:19:37.961 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-30T07:19:39.719 INFO:tasks.workunit.client.0.smithi047.stdout:StatECPP_smithi047-41102-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi047-41102-7"}]: dispatch 2024-03-30T07:19:39.719 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:26.676920+0000 mon.c [INF] from='client.? 172.21.15.47:0/3653961619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi047-41311-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:19:39.719 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:27.077320+0000 mon.a [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-03-30T07:19:39.719 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:27.131982+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi047-40956-4"}]: dispatch 2024-03-30T07:19:39.719 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:27.144267+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:27.148550+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi047-41102-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi047-41102-7"}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:28.161237+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi047-41311-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:28.726275+0000 mon.a [WRN] Health check update: 0 slow ops, oldest one blocked for 55 sec, daemons [osd.1,osd.2] have slow ops. (SLOW_OPS) 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:29.140584+0000 mon.b [INF] from='client.? 172.21.15.47:0/2416272721' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:29.144780+0000 mon.b [INF] from='client.? 172.21.15.47:0/3381790431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi047-40753-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:31.412318+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 17311.090ms) (OSD_SLOW_PING_TIME_BACK) 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:31.415299+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 15854.751ms) (OSD_SLOW_PING_TIME_FRONT) 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:31.415751+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 55 sec, osd.2 has slow ops) 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:31.628597+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:31.634887+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi047-40753-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:32.361354+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi047-41225-6", "pg_num": 4}]': finished 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:32.363185+0000 mon.c [INF] from='client.? 172.21.15.47:0/2748660152' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T07:19:39.720 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:32.366464+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi047-40956-4"}]': finished 2024-03-30T07:19:42.032 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:09.843 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:42.047 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:09.859 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:42.331 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:04:10.140 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:42.968 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:10.746 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:42.974 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:10.752 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:43.040 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:10.852 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:43.054 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:10.865 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:43.154 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:04:10.932 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:32.387694+0000 mon.a [INF] from='client.? ' ent api_list: [==========] Running 11 tests from 3 test suites. 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] Global test environment set-up. 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.ListObjects (86982 ms) 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (3933 ms) 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo1 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo2 2024-03-30T07:19:43.198 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo3 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo4 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo5 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo1 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo6,foo7 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo7 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo6 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo4 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo5 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns2:foo7 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns2:foo6 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo1 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo1 2024-03-30T07:19:43.199 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo2 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo3 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (7867 ms) 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 1 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 10 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 13 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 7 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 14 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 0 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 15 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 11 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 5 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 8 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 6 0 2024-03-30T07:19:43.200 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 3 0 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 4 0 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 12 0 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 9 0 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 2 0 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (7049 ms) 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: x cursor=MIN 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-03-30T07:19:43.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-03-30T07:19:43.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:566253c9:::13:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-03-30T07:19:43.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-03-30T07:19:43.282 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:04:11.093 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:19:47.532 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:863748b0:::15:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-03-30T07:19:47.533 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-03-30T07:19:47.534 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:e960b815:::9:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-03-30T07:19:47.535 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=10 expected=10 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=0 expected=0 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=7 expected=7 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:566253c9:::13:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=13 expected=13 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-03-30T07:19:47.536 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=8 expected=8 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=5 expected=5 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=11 expected=11 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:863748b0:::15:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-03-30T07:19:47.537 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=15 expected=15 2024-03-30T07:19:47.538 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-03-30T07:19:54.123 INFO:tasks.workunit.client.0.smithi047.stdout: ity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi047-41311-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:34.151533+0000 client.admin [INF] onexx 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:34.295623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:37.462337+0000 mon.b [INF] from='client.? 172.21.15.47:0/3381790431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi047-40753-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:39.876000+0000 mon.c [INF] from='client.? 172.21.15.47:0/2748660152' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.192220+0000 mon.a [INF] from='client.? 172.21.15.47:0/2842640767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi047-41151-7"}]: dispatch 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.288936+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi047-40753-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi047-40753-22"}]: dispatch 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.390535+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi047-40956-4", "tierpool": "test-rados-api-smithi047-40956-4-cache"}]': finished 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.398881+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi047-41225-4", "tierpool": "test-rados-api-smithi047-41225-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.401629+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi047-40823-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T07:19:54.124 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:40.655452+0000 mon.b [INF] from='client.? 172.21.15.47:0/2841959700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.151270+0000 mon.a [INF] from='client.? 172.21.15.47:0/2842640767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi047-41151-7"}]: dispatch 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.438621+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.576774+0000 mon.a [INF] from='client.? 172.21.15.47:0/2842640767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi047-41151-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.959514+0000 client.admin [INF] twoxx 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.995873+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-30T07:19:54.125 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.328662+0000 mon.b [INF] from='client.? 172.21.15.47:0/3562570157' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi047-41225-4", "overlaypool": "test-rados-api-smithi047-41225-6"}]: dispatch 2024-03-30T07:19:57.955 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.332696+0000 mon.b [INF] from='client.? 172.21.15.47:0/2416272721' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi047-40823-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:41.453132+0000 mon.b [INF] from='client.12532 172.21.15.47:0/2841959700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:42.260733+0000 mon.c [INF] from='client.? 172.21.15.47:0/1213844353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi047-40733-16"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:43.953501+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi047-41225-4", "overlaypool": "test-rados-api-smithi047-41225-6"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:43.962031+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi047-40823-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi047-40823-10"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:43.970946+0000 mon.a [INF] from='client.12532 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:44.041224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi047-40733-16"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:49.192108+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:45.305994+0000 mon.b [INF] from='client.12532 172.21.15.47:0/2841959700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:47.666623+0000 mon.b [INF] from='client.? 172.21.15.47:0/2263546994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi047-41196-12"}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:48.860053+0000 mon.c [INF] from='client.? 172.21.15.47:0/1213844353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi047-40733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:49.198425+0000 mon.c [INF] from='client.? 172.21.15.47:0/2443661348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi047-40695-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:19:57.956 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:49.573041+0000 mon.c [INF] from='client.? 172.21.15.47:0/1271782773' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi047-40714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:19:57.957 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.063951+0000 mon.a [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T07:19:57.957 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.528080+0000 mon.a [INF] from='client.12532 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi047-40797-10"}]: dispatch 2024-03-30T07:19:57.957 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.566749+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi047-41196-12"}]: dispatch 2024-03-30T07:20:01.709 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.699768+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi047-40733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:20:01.709 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.818772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi047-40753-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi047-40753-22"}]': finished 2024-03-30T07:20:01.709 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.823362+0000 mon.a [INF] from='client.? 172.21.15.47:0/2842640767' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi047-41151-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T07:20:01.709 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.825929+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi047-41225-4", "overlaypool": "test-rados-api-smithi047-41225-6"}]': finished 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.942021+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi047-40695-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:50.977606+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi047-40714-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:51.335678+0000 mon.a [INF] from='client.? 172.21.15.47:0/2842640767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi047-41151-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi047-41151-7"}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:51.934344+0000 mon.b [INF] from='client.? 172.21.15.47:0/3562570157' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi047-41225-6", "mode": "writeback"}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:51.945338+0000 mon.b [INF] from='client.12532 172.21.15.47:0/2841959700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi047-40797-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:51.948064+0000 mon.b [INF] from='client.? 172.21.15.47:0/2263546994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi047-41196-12"}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:52.347317+0000 mon.c [INF] from='client.? 172.21.15.47:0/2748660152' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T07:20:01.710 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:53.957893+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi047-41225-6", "mode": "writeback"}]: dispatch 2024-03-30T07:20:01.711 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:53.967928+0000 mon.a [INF] from='client.12532 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi047-40797-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T07:20:01.711 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:53.971803+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi047-41196-12"}]: dispatch 2024-03-30T07:20:01.711 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.560268+0000 client.admin [INF] threexx 2024-03-30T07:20:08.578 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.623590+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi047-40823-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testp api_io_pp: Running main() from gmock_main.cc 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: seed 40753 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (75969 ms) 2024-03-30T07:20:08.579 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (1338 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (66 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (316 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (758 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (142 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (40 ms) 2024-03-30T07:20:08.580 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (70 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (1286 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (57 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (88 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (83 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (1240 ms) 2024-03-30T07:20:08.581 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (1445 ms) 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (315 ms) 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (169 ms) 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (555 ms) 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (157 ms) 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-30T07:20:08.582 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (130 ms) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (84224 ms total) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6034 ms) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (1270 ms) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (382 ms) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (6335 ms) 2024-03-30T07:20:08.583 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (21 api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (26466 ms) 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (27742 ms) 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (54208 ms total) 2024-03-30T07:20:13.030 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (29981 ms) 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (44567 ms) 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (512 ms) 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (136 ms) 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435266 notifier_gid 4807 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (212 ms) 2024-03-30T07:20:13.031 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435267 notifier_gid 4807 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (202 ms) 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435268 notifier_gid 4807 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (366 ms) 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447601664 notify_id 468151435264 notifier_gid 4807 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (207 ms) 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435265 notifier_gid 4807 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (255 ms) 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435269 notifier_gid 4807 2024-03-30T07:20:13.032 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (205 ms) 2024-03-30T07:20:13.033 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-30T07:20:13.033 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: trying... 2024-03-30T07:20:13.033 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 468151435270 notifier_gid 4807 2024-03-30T07:20:13.033 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: timed out 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flush api_misc: [----------] 8 tests from LibRadosMisc 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.Exec (30259 ms) 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-30T07:20:18.528 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (1673 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (1680 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6021 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (69809 ms) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] 8 tests from LibRadosMisc (109442 ms total) 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [----------] Global test environment tear-down 2024-03-30T07:20:18.529 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (148193 ms total) 2024-03-30T07:20:18.530 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout:rofile-LibRadosLockECPP_smithi047-40823-10"}]': finished 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.630020+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi047-40733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.634382+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi047-40695-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.638988+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi047-40714-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:54.781905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: got: 2024-03-30T07:19:55.610260+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (48821 ms) 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (148374 ms total) 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (148374 ms total) 2024-03-30T07:20:18.916 INFO:tasks.workunit.client.0.smithi047.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: seed 41102 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (7673 ms) 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (11061 ms) 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-30T07:20:19.525 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (35 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (10951 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (42896 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (72616 ms total) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (5299 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (4316 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1481 ms) 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-30T07:20:19.526 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16498 ms) 2024-03-30T07:20:19.527 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (27594 ms total) 2024-03-30T07:20:19.527 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: 2024-03-30T07:20:19.527 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-30T07:20:19.527 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (149081 ms total) 2024-03-30T07:20:19.527 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-30T07:20:19.856 INFO:tasks.workunit.client.0.smithi047.stdout:2 ms) 2024-03-30T07:20:19.856 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-30T07:20:19.856 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (239 ms) 2024-03-30T07:20:19.856 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-30T07:20:19.856 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (608 ms) 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (136 ms) 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (226 ms) 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (334 ms) 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (130 ms) 2024-03-30T07:20:19.857 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (278 ms) 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (740 ms) 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (205 ms) 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (1969 ms) 2024-03-30T07:20:19.858 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (129 ms) 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (237 ms) 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (19464 ms total) 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (149579 ms total) 2024-03-30T07:20:19.859 INFO:tasks.workunit.client.0.smithi047.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: Running main() from gmock_main.cc 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] Global test environment set-up. 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-30T07:20:25.463 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStat.Stat (53069 ms) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6734 ms) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStat.StatNS (28848 ms) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (2345 ms) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3304 ms) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] 5 tests from LibRadosStat (94300 ms total) 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-30T07:20:25.464 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (15184 ms) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (4896 ms) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (73 ms) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3161 ms) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (23314 ms total) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [----------] Global test environment tear-down 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (155007 ms total) 2024-03-30T07:20:25.465 INFO:tasks.workunit.client.0.smithi047.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-30T07:20:27.037 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (42733 ms) 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93957132070448 err -107 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (2589 ms) 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-30T07:20:27.038 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93957132070448 err -107 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (20131 ms) 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 382252089344 cookie 93957133635584 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (8319 ms) 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-30T07:20:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 420906795008 cookie 93957133635584 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (12336 ms) 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 420906795009 cookie 93957133635584 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (3281 ms) 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-30T07:20:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 425201762305 cookie 93957133635584 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 425201762305 cookie 93957133660736 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (2554 ms) 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 429496729601 cookie 93957133660736 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 429496729602 cookie 93957133660736 2024-03-30T07:20:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 429496729602 cookie 93957133660736 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 429496729603 cookie 93957133660736 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (4180 ms) 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93957133660736 err -107 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 438086664195 cookie 93957133660736 2024-03-30T07:20:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_cb from 4549 notify_id 438086664196 cookie 93957133660736 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5416 ms) 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93957133660736 err -107 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (264 ms) 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (101803 ms total) 2024-03-30T07:20:27.043 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: 2024-03-30T07:20:27.044 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-30T07:20:27.044 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-30T07:20:27.044 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: watch_notify_test_cb 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (8597 ms) 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (8597 ms total) 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (159659 ms total) 2024-03-30T07:20:30.129 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-30T07:20:30.983 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (37207 ms) 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (18879 ms) 2024-03-30T07:20:30.984 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (13813 ms) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (15227 ms) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (33703 ms) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (36075 ms) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5556 ms) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (160460 ms total) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (160460 ms total) 2024-03-30T07:20:30.985 INFO:tasks.workunit.client.0.smithi047.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout:ing 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flushed 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3113 ms) 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: trying... 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 472446402567 notifier_gid 4807 2024-03-30T07:20:37.105 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: timed out 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flushing 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flushed 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3117 ms) 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: List watches 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify2 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 498216206344 notifier_gid 4807 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify2 done 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: watch_check 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: unwatch2 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flushing 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: done 2024-03-30T07:20:37.106 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (10475 ms) 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: List watches 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify2 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: handle_notify cookie 94239447587136 notify_id 523986010121 notifier_gid 4807 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: notify2 done 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: watch_check 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: unwatch2 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: flushing 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: done 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (5704 ms) 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (99052 ms total) 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (166617 ms total) 2024-03-30T07:20:37.107 INFO:tasks.workunit.client.0.smithi047.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: seed 40823 2024-03-30T07:20:39.182 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (39089 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (5402 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (17049 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4591 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5395 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (2189 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (3682 ms) 2024-03-30T07:20:39.183 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (893 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (78290 ms total) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (7590 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (187 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (17103 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (3512 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3982 ms) 2024-03-30T07:20:39.184 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (2616 ms) 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (1748 ms) 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (1899 ms) 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (38637 ms total) 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (168851 ms total) 2024-03-30T07:20:39.185 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-30T07:20:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: starting. 2024-03-30T07:20:41.757 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: creating pool ceph_test_rados_list_parallel.smithi047-41433 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: created object 0... 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: created object 25... 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: created object 49... 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: finishing. 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_1_[41482]: shutting down. 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: starting. 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: listing objects. 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: listed object 0... 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: listed object 25... 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: saw 50 objects 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_2_[41484]: shutting down. 2024-03-30T07:20:41.758 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: starting. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: creating pool ceph_test_rados_list_parallel.smithi047-41433 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: created object 0... 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: created object 25... 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: created object 49... 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: finishing. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_3_[68059]: shutting down. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: starting. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: listing objects. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: listed object 0... 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: listed object 25... 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: saw 43 objects 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_4_[68060]: shutting down. 2024-03-30T07:20:41.759 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: starting. 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: removed 25 objects... 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: removed half of the objects 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: removed 50 objects... 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: removed 50 objects 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_5_[68061]: shutting down. 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: starting. 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: creating pool ceph_test_rados_list_parallel.smithi047-41433 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: created object 0... 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: created object 25... 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: created object 49... 2024-03-30T07:20:41.760 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: finishing. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_6_[77900]: shutting down. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: starting. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: listing objects. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: listed object 0... 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: listed object 25... 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: listed object 50... 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: saw 54 objects 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_7_[77901]: shutting down. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: starting. 2024-03-30T07:20:41.761 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: added 25 objects... 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: added half of the objects 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: added 50 objects... 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: added 50 objects 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_8_[77902]: shutting down. 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:41.762 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ********************** api_lock: Running main() from gmock_main.cc 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] Global test environment set-up. 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (31276 ms) 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.LockShared (54975 ms) 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-30T07:20:45.201 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (2488 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1114 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (439 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.Unlock (2591 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (209 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (96 ms) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] 8 tests from LibRadosLock (93188 ms total) 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-30T07:20:45.202 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (7999 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (3538 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (3462 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (2254 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3853 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (3296 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (1203 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (129 ms) 2024-03-30T07:20:45.203 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (25734 ms total) 2024-03-30T07:20:45.204 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: 2024-03-30T07:20:45.204 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [----------] Global test environment tear-down 2024-03-30T07:20:45.204 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (174903 ms total) 2024-03-30T07:20:45.204 INFO:tasks.workunit.client.0.smithi047.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-30T07:20:45.277 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:13.088 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.317 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:13.128 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.328 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:13.139 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.336 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:13.147 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.382 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:13.160 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.384 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:13.161 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.547 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:13.358 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.577 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:13.355 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:45.604 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:13.415 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:20:49.060 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: starting. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: creating pool ceph_test_rados_delete_pools_parallel.smithi047-41489 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: created object 0... 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: created object 25... 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: created object 49... 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: finishing. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_1_[41512]: shutting down. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_2_[41513]: starting. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_2_[41513]: deleting pool ceph_test_rados_delete_pools_parallel.smithi047-41489 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_2_[41513]: shutting down. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******************************* 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: starting. 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: creating pool ceph_test_rados_delete_pools_parallel.smithi047-41489 2024-03-30T07:20:49.061 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: created object 0... 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: created object 25... 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: created object 49... 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: finishing. 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_3_[69739]: shutting down. 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******************************* 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: starting. 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: listing objects. 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: listed object 0... 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: listed object 25... 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: saw 50 objects 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_5_[69742]: shutting down. 2024-03-30T07:20:49.062 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******************************* 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_4_[69740]: starting. 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_4_[69740]: deleting pool ceph_test_rados_delete_pools_parallel.smithi047-41489 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: process_4_[69740]: shutting down. 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******************************* 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******************************* 2024-03-30T07:20:49.063 INFO:tasks.workunit.client.0.smithi047.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: starting. 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: creating pool ceph_test_rados_open_pools_parallel.smithi047-41502 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: created object 0... 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: created object 25... 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: created object 49... 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: finishing. 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_1_[41535]: shutting down. 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_2_[41536]: starting. 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_2_[41536]: rados_pool_create. 2024-03-30T07:20:49.084 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_2_[41536]: rados_ioctx_create. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_2_[41536]: shutting down. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: ******************************* 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: starting. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: creating pool ceph_test_rados_open_pools_parallel.smithi047-41502 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: created object 0... 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: created object 25... 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: created object 49... 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: finishing. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_3_[79531]: shutting down. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: ******************************* 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_4_[79532]: starting. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_4_[79532]: rados_pool_create. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_4_[79532]: rados_ioctx_create. 2024-03-30T07:20:49.085 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: process_4_[79532]: shutting down. 2024-03-30T07:20:49.086 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: ******************************* 2024-03-30T07:20:49.086 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: ******************************* 2024-03-30T07:20:49.086 INFO:tasks.workunit.client.0.smithi047.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-30T07:20:50.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: Running main() from gmock_main.cc 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] Global test environment set-up. 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (33395 ms) 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (27859 ms) 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6342 ms) 2024-03-30T07:20:50.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.Checksum (17515 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (1273 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (153 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (360 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (315 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.TruncTest (84 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (105 ms) 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-30T07:20:50.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2486 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.RmXattr (387 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIo.XattrIter (127 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] 14 tests from LibRadosIo (90402 ms total) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (9129 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4389 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (2733 ms) 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-30T07:20:50.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1109 ms) 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (3518 ms) 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3966 ms) 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (372 ms) 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (632 ms) 2024-03-30T07:20:50.439 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2229 ms) 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (1071 ms) 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] 10 tests from LibRadosIoEC (29148 ms total) 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [----------] Global test environment tear-down 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [==========] 24 tests from 2 test suites ran. (180166 ms total) 2024-03-30T07:20:50.440 INFO:tasks.workunit.client.0.smithi047.stdout: api_io: [ PASSED ] 24 tests. 2024-03-30T07:21:05.472 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:33.283 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.479 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:33.290 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.534 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:33.312 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.536 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:33.313 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.540 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:33.352 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.542 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:33.353 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.556 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:33.367 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.590 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:33.368 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:05.619 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:33.431 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:06.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (43387 ms) 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4483 ms) 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (9964 ms) 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5868 ms) 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (63702 ms total) 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: 2024-03-30T07:21:06.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (25558 ms) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (39506 ms) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (23045 ms) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (88109 ms total) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4599 ms) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1835 ms) 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-30T07:21:06.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2308 ms) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4733 ms) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (13475 ms total) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4328 ms) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (5534 ms) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (9862 ms total) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (196159 ms total) 2024-03-30T07:21:06.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout:********* 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: starting. 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: creating pool ceph_test_rados_list_parallel.smithi047-41433 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: created object 0... 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: created object 25... 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: created object 49... 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: finishing. 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_9_[103027]: shutting down. 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.551 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: starting. 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: listing objects. 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: listed object 0... 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: listed object 25... 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: listed object 50... 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: listed object 75... 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: saw 96 objects 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_10_[103028]: shutting down. 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: starting. 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: removed 25 objects... 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: removed half of the objects 2024-03-30T07:21:23.552 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: removed 50 objects... 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: removed 50 objects 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_13_[103031]: shutting down. 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: starting. 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: added 25 objects... 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: added half of the objects 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: added 50 objects... 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: added 50 objects 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_12_[103030]: shutting down. 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.553 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: starting. 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: added 25 objects... 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: added half of the objects 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: added 50 objects... 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: added 50 objects 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_11_[103029]: shutting down. 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: starting. 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: creating pool ceph_test_rados_list_parallel.smithi047-41433 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: created object 0... 2024-03-30T07:21:23.554 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: created object 25... 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: created object 49... 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: finishing. 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_14_[105086]: shutting down. 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: starting. 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listing objects. 2024-03-30T07:21:23.555 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 0... 2024-03-30T07:21:25.481 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:53.292 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.483 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:53.294 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.556 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:53.367 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.557 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:53.334 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.557 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:53.369 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.559 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:53.337 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.582 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:05:53.393 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.620 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:05:53.431 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:25.625 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:05:53.403 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:29.364 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 25... 2024-03-30T07:21:29.364 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 50... 2024-03-30T07:21:29.364 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 75... 2024-03-30T07:21:29.364 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 100... 2024-03-30T07:21:29.364 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: listed object 125... 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: saw 150 objects 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_15_[105087]: shutting down. 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: starting. 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: added 25 objects... 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: added half of the objects 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: added 50 objects... 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: added 50 objects 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: process_16_[105088]: shutting down. 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.365 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.366 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.366 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******************************* 2024-03-30T07:21:29.366 INFO:tasks.workunit.client.0.smithi047.stdout: list_parallel: ******* SUCCESS ********** 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: seed 41034 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (36567 ms) 2024-03-30T07:21:32.676 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (9336 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (40216 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (12458 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (31246 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (129823 ms total) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (9281 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (8309 ms) 2024-03-30T07:21:32.677 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (11958 ms) 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5318 ms) 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3026 ms) 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1055-g55c4b44e/rpm/el9/BUILD/ceph-18.2.2-1055-g55c4b44e/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi047-41034-7 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18126 ms) 2024-03-30T07:21:32.678 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (56018 ms total) 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5347 ms) 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5354 ms total) 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4572 ms) 2024-03-30T07:21:32.679 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-30T07:21:45.697 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:13.508 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.702 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:13.513 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.705 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:13.482 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.707 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:13.518 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.722 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:13.532 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.750 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:13.527 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.790 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:13.568 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.825 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:13.636 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:21:45.835 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:13.646 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:04.876 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:04.874+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:05.676 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:33.487 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.692 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:33.503 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.717 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:33.528 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.724 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:33.535 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.727 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:33.504 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.729 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:33.540 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.730 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:33.508 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.752 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:33.529 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.755 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:33.566 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:05.840 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:05.837+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:06.885 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:06.881+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:07.919 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:07.916+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:08.950 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:08.947+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:09.921 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:09.916+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:10.966 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:10.962+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:11.976 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:11.971+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:12.973 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:12.969+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:13.969 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:13.966+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:15.005 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:15.002+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:15.982 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:22:15.979+0000 ea58640 -1 osd.1 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21883.0:13 110.6s0 110:659ac1b9:test-rados-api-smithi047-41034-20::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-03-30T07:22:25.752 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:53.563 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.756 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:53.567 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.802 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:53.580 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.806 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:53.617 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.806 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:53.584 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.811 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:53.622 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.815 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:06:53.626 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.842 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:06:53.619 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:25.848 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:06:53.659 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:27.283 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] Lib api_aio_pp: Running main() from gmock_main.cc 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4135 ms) 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (47144 ms) 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (20123 ms) 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (8146 ms) 2024-03-30T07:22:27.284 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (23247 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (40069 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (11172 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (5700 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (4482 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3636 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2951 ms) 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-30T07:22:27.285 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (5605 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (5630 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4008 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3544 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2652 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4011 ms) 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-30T07:22:27.286 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3053 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2918 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4063 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3978 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4291 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3481 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4380 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (46258 ms) 2024-03-30T07:22:27.287 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-30T07:22:27.288 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3428 ms) 2024-03-30T07:22:27.288 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-30T07:22:45.863 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:13.673 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.867 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:13.679 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.875 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:13.652 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.876 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:13.654 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.931 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:13.742 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.940 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:13.751 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.984 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:13.761 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.993 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:13.803 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:22:45.994 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:13.804 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:03.239 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:03.236+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:04.268 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:04.265+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:05.277 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:05.275+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:05.867 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:33.678 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.871 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:33.682 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.911 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:33.722 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.920 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:33.697 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.922 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:33.700 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.932 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:33.743 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.935 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:33.747 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.944 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:33.721 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:05.959 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:33.770 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:06.290 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:06.287+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:07.313 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:07.311+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:07.849 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:07.845+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:08.229 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:36.040 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.239 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:36.050 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.256 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:36.067 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.282 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:36.060 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.283 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:36.061 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.290 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:36.101 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.291 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:36.102 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.298 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:36.076 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.305 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:36.117 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:08.348 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:08.346+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:08.896 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:08.893+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:09.389 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:09.385+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:09.893 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:09.888+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:10.349 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:10.346+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:10.925 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:10.922+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:11.350 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:11.348+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:11.909 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:11.906+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:12.324 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:12.321+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:12.872 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:12.869+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:13.370 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:13.365+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:13.849 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:13.845+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:14.377 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:14.375+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:14.815 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:14.812+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:15.364 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:15.359+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:15.840 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:15.836+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:16.321 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:16.317+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:16.796 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:16.794+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:17.327 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:17.324+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:17.830 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:17.828+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:18.293 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:18.291+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:18.821 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:18.818+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:19.324 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:19.321+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:19.665 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:19.662+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:19.780 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:19.776+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:20.372 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:20.369+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:20.658 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:20.655+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:20.767 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:20.764+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:21.404 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:21.400+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:21.705 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:21.702+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:21.792 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:21.790+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:22.384 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:22.380+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:22.693 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:22.690+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:22.848 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:22.844+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:23.379 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:23.375+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:23.666 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:23.664+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:23.849 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:23.844+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:24.362 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:24.358+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:24.629 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:24.627+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:24.834 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:24.827+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:25.364 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:25.361+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:25.653 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:25.650+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:25.795 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:25.792+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:25.923 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:53.733 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.927 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:53.738 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.952 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:07:53.763 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.975 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:53.752 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.976 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:53.754 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.978 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:53.789 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:25.981 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:53.792 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:26.002 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:07:53.779 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:26.007 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:07:53.819 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:26.348 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:26.346+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:26.670 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:26.667+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:26.819 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:26.815+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:27.344 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:27.339+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:27.632 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:27.629+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:27.814 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:27.810+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:28.345 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:28.343+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:28.658 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:28.656+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:28.768 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:28.765+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:29.371 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:29.369+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:29.679 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:29.676+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:29.813 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:29.810+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:30.344 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:30.340+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:30.701 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:30.699+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:30.799 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:30.795+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:31.344 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:31.341+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:31.734 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:31.731+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:31.784 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:31.780+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:32.390 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:32.387+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:32.743 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:32.740+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:32.766 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:32.764+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:33.447 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:33.442+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:33.728 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:33.726+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:33.738 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:33.735+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:34.497 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:34.492+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:34.716 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:34.713+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:34.759 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:34.756+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:35.535 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:35.532+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:35.680 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:35.677+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:35.711 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:35.709+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:36.533 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:36.529+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:36.667 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:36.665+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:36.671 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:36.669+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:37.553 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:37.551+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:37.678 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:37.676+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:37.713 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:37.711+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:38.611 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:38.606+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:38.677 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:38.675+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:38.701 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:38.698+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:39.566 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:39.563+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:39.704 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:39.701+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:39.705 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:39.702+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:40.558 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:40.555+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:40.675 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:40.672+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:40.720 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:40.717+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:41.565 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:41.563+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:41.712 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:41.709+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:41.741 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:41.739+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:42.583 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:42.580+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:42.756 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:42.754+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:42.761 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:42.758+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:43.594 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:43.591+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:43.798 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:43.794+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:43.804 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:43.802+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:44.586 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:44.583+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:44.778 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:44.775+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:44.830 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:44.827+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:45.630 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:45.626+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:45.795 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:45.792+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:45.874 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:45.871+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:46.184 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:13.995 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.186 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:13.997 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.212 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:14.023 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.231 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:14.009 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.234 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:14.045 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.235 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:14.012 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.238 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:14.049 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.262 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:14.039 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.266 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:14.077 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:23:46.616 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:46.613+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:46.830 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:46.828+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:46.831 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:46.827+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:47.646 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:47.643+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:47.836 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:47.834+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:47.863 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:47.860+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:48.649 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:48.647+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:48.840 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:48.838+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:48.906 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:48.905+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:49.668 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:49.667+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:49.850 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:49.848+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:49.915 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:49.912+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:50.641 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:50.639+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:50.877 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:50.875+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:50.939 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:50.936+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:51.683 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:51.681+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:51.858 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:51.855+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:51.955 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:51.953+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:52.730 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:52.728+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:52.878 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:52.877+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:52.993 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:52.984+0000 ea58640 -1 osd.6 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.326012+0000 front 2024-03-30T07:22:29.322215+0000 (oldest deadline 2024-03-30T07:23:52.812328+0000) 2024-03-30T07:23:52.998 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:52.995+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:53.706 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:53.703+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:53.822 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:23:53.815+0000 12a60640 -1 osd.2 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.754899+0000 front 2024-03-30T07:22:29.789079+0000 (oldest deadline 2024-03-30T07:23:53.232758+0000) 2024-03-30T07:23:53.895 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:53.892+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:53.989 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:53.987+0000 ea58640 -1 osd.6 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.326012+0000 front 2024-03-30T07:22:29.322215+0000 (oldest deadline 2024-03-30T07:23:52.812328+0000) 2024-03-30T07:23:53.990 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:53.988+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:54.758 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:54.745+0000 12a60640 -1 osd.4 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:31.915045+0000 front 2024-03-30T07:22:31.845767+0000 (oldest deadline 2024-03-30T07:23:53.985454+0000) 2024-03-30T07:23:54.767 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:54.764+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:54.792 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:23:54.789+0000 12a60640 -1 osd.2 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.754899+0000 front 2024-03-30T07:22:29.789079+0000 (oldest deadline 2024-03-30T07:23:53.232758+0000) 2024-03-30T07:23:54.867 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:54.864+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:54.995 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:54.993+0000 ea58640 -1 osd.6 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.326012+0000 front 2024-03-30T07:22:29.322215+0000 (oldest deadline 2024-03-30T07:23:52.812328+0000) 2024-03-30T07:23:54.996 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:54.994+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:55.206 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:23:55.199+0000 12a60640 -1 osd.0 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.984744+0000 front 2024-03-30T07:22:29.974441+0000 (oldest deadline 2024-03-30T07:23:54.656990+0000) 2024-03-30T07:23:55.794 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:55.792+0000 12a60640 -1 osd.4 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:31.915045+0000 front 2024-03-30T07:22:31.845767+0000 (oldest deadline 2024-03-30T07:23:53.985454+0000) 2024-03-30T07:23:55.795 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:55.793+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:55.841 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:23:55.838+0000 12a60640 -1 osd.2 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.754899+0000 front 2024-03-30T07:22:29.789079+0000 (oldest deadline 2024-03-30T07:23:53.232758+0000) 2024-03-30T07:23:55.859 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:55.853+0000 12a60640 -1 osd.7 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:30.433436+0000 front 2024-03-30T07:22:30.360212+0000 (oldest deadline 2024-03-30T07:23:55.280702+0000) 2024-03-30T07:23:55.859 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:55.857+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:55.950 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:55.945+0000 ea58640 -1 osd.6 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.326012+0000 front 2024-03-30T07:22:29.322215+0000 (oldest deadline 2024-03-30T07:23:52.812328+0000) 2024-03-30T07:23:55.956 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:55.953+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:56.157 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:23:56.154+0000 12a60640 -1 osd.0 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.984744+0000 front 2024-03-30T07:22:29.974441+0000 (oldest deadline 2024-03-30T07:23:54.656990+0000) 2024-03-30T07:23:56.777 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:56.772+0000 12a60640 -1 osd.4 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:31.915045+0000 front 2024-03-30T07:22:31.845767+0000 (oldest deadline 2024-03-30T07:23:53.985454+0000) 2024-03-30T07:23:56.782 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:56.779+0000 12a60640 -1 osd.4 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:56.833 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:23:56.824+0000 12a60640 -1 osd.2 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.754899+0000 front 2024-03-30T07:22:29.789079+0000 (oldest deadline 2024-03-30T07:23:53.232758+0000) 2024-03-30T07:23:56.899 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:56.890+0000 12a60640 -1 osd.7 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:30.433436+0000 front 2024-03-30T07:22:30.360212+0000 (oldest deadline 2024-03-30T07:23:55.280702+0000) 2024-03-30T07:23:56.906 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:56.904+0000 12a60640 -1 osd.7 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:56.999 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:56.989+0000 ea58640 -1 osd.6 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.326012+0000 front 2024-03-30T07:22:29.322215+0000 (oldest deadline 2024-03-30T07:23:52.812328+0000) 2024-03-30T07:23:57.000 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:56.995+0000 ea58640 -1 osd.6 216 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:57.117 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:23:57.110+0000 12a60640 -1 osd.0 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.984744+0000 front 2024-03-30T07:22:29.974441+0000 (oldest deadline 2024-03-30T07:23:54.656990+0000) 2024-03-30T07:23:57.363 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:23:57.356+0000 ea58640 -1 osd.1 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:31.082931+0000 front 2024-03-30T07:22:31.074259+0000 (oldest deadline 2024-03-30T07:23:56.870247+0000) 2024-03-30T07:23:57.828 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:23:57.826+0000 12a60640 -1 osd.4 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:384 77.1f 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) 2024-03-30T07:23:57.936 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:57.935+0000 12a60640 -1 osd.7 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:23:58.010 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:23:58.006+0000 ea58640 -1 osd.6 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21970.0:2 126.13 126.7c099dd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e210) 2024-03-30T07:23:58.087 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:23:58.084+0000 12a60640 -1 osd.0 216 heartbeat_check: no reply from 172.21.15.52:6830 osd.5 since back 2024-03-30T07:22:29.984744+0000 front 2024-03-30T07:22:29.974441+0000 (oldest deadline 2024-03-30T07:23:54.656990+0000) 2024-03-30T07:23:58.951 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:23:58.927+0000 12a60640 -1 osd.7 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:24:00.068 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:24:00.063+0000 12a60640 -1 osd.7 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4498.0:2078 11.1f 11:ffee52ae:test-rados-api-smithi047-40904-17::big.copy:head [copy-from ver 0 in=74b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216) 2024-03-30T07:24:03.573 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRad api_aio: Running main() from gmock_main.cc 2024-03-30T07:24:03.573 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-30T07:24:03.573 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] Global test environment set-up. 2024-03-30T07:24:03.573 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4220 ms) 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (34369 ms) 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (9174 ms) 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (7508 ms) 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (48090 ms) 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-30T07:24:03.574 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (39326 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (11350 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (6665 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4046 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (8290 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (7965 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3147 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (13090 ms) 2024-03-30T07:24:03.575 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3953 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.Flush (4126 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4179 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4001 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4538 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4379 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (46274 ms) 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-30T07:24:03.576 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3442 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (10509 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3321 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3374 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (79886 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4088 ms) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] 26 tests from LibRadosAio (373310 ms total) 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: 2024-03-30T07:24:03.577 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-30T07:24:04.750 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpRadosSnapshotsECPP.SnapRemovePP (3313 ms) 2024-03-30T07:24:04.750 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2104 ms) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (43154 ms) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (53143 ms total) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5145 ms) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4007 ms) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (87489 ms) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (96641 ms total) 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: 2024-03-30T07:24:04.751 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-30T07:24:04.752 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (374338 ms total) 2024-03-30T07:24:04.752 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-30T07:24:04.752 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-30T07:24:04.752 INFO:tasks.workunit.client.0.smithi047.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-30T07:24:06.337 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:34.148 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.345 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:34.155 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.345 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:34.122 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.348 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:34.159 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.350 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:34.127 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.353 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:34.164 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.369 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:24:06.358+0000 ea58640 -1 osd.5 205 get_health_metrics reporting 1 slow ops, oldest is osd_repop(client.21706.0:384 77.1f e205/112 77:fd18e0bd:test-rados-api-smithi047-40956-14::foo15:head v 205'6, mlcod=169'3) 2024-03-30T07:24:06.455 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:34.232 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.478 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:34.288 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:06.514 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:34.325 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:11.126 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:24:11.122+0000 1025b640 -1 osd.5 228 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:24:26.456 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:54.266 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.508 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:54.318 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.524 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:54.301 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.526 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:54.304 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.559 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:08:54.370 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.564 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:54.375 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.566 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:54.377 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.595 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:08:54.372 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:26.624 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:08:54.435 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: seed 40904 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-30T07:24:44.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (61905 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (11826 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (3000 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (1359 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (298 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (45 ms) 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-30T07:24:44.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (110 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (57 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (249 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (58 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (87 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (163 ms) 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-30T07:24:44.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (145577 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (5365 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: done waiting, doing copies 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: done waiting 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (174171 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (379 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (46 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4462 ms) 2024-03-30T07:24:44.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-30T07:24:44.718 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-30T07:24:44.718 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-30T07:24:44.718 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-30T07:24:44.718 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-30T07:24:44.718 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-30T07:24:46.678 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:14.489 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.682 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:14.493 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.744 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.521 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.746 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.523 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.751 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:14.562 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.754 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:14.565 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.759 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:14.569 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.801 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:14.578 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:24:46.826 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:14.637 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.126 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:32.937 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.141 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:32.952 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.158 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:32.969 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.160 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:32.937 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.161 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:32.972 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.164 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:32.941 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.292 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:33.101 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.293 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:33.070 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:05.299 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:33.109 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.677 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:34.487 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.739 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:34.550 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.742 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:34.519 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.744 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:34.521 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.756 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:34.533 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.763 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:34.573 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.815 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:34.626 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.823 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:34.634 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:06.839 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:34.650 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:24.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (409159 ms total) 2024-03-30T07:25:24.435 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2943 ms) 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2943 ms total) 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (543 ms) 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (55 ms) 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (598 ms total) 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-30T07:25:24.436 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (617 ms) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (63 ms) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (680 ms total) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (405 ms) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (42 ms) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (447 ms total) 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-30T07:25:24.437 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (12514 ms) 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (12514 ms total) 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (454057 ms total) 2024-03-30T07:25:24.438 INFO:tasks.workunit.client.0.smithi047.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-30T07:25:26.803 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:54.614 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.806 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:54.617 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.855 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:54.632 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.856 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:09:54.666 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.863 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:54.640 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.872 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:54.683 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.876 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:54.687 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.907 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:09:54.684 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:26.936 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:09:54.747 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.596 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:51.592+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:51.781 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:19.591 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.784 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:19.595 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.829 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:19.640 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.832 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:19.609 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.835 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:19.613 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.840 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:19.651 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.850 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:19.660 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.874 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:19.651 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:51.887 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:19.698 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:25:52.580 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:52.577+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:53.583 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:53.580+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:54.620 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:54.618+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:55.573 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:55.570+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:56.589 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:56.586+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:57.625 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:57.622+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:58.614 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:58.611+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:25:58.706 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:25:58.698+0000 12a60640 -1 osd.4 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16892.0:1 158.7s0 158.607d4ce7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e286) 2024-03-30T07:25:59.569 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:25:59.567+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:26:00.303 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:26:00.298+0000 12a60640 -1 osd.4 312 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16892.0:1 158.7s0 158.607d4ce7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e286) 2024-03-30T07:26:00.583 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:26:00.581+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:26:01.320 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:26:01.287+0000 12a60640 -1 osd.4 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16892.0:1 158.7s0 158:e732be06:::oid_WaitForSafePP_:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286) 2024-03-30T07:26:01.581 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:26:01.576+0000 12a60640 -1 osd.0 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21706.0:976 117.1 117:87baf313:test-rados-api-smithi047-40956-14::foo7:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e298) 2024-03-30T07:26:11.931 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:39.741 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.934 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:39.745 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.969 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:39.779 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.986 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:39.763 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.987 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:39.764 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.991 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:39.801 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:11.993 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:39.804 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:12.018 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:39.795 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:12.030 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:39.840 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:27.039 INFO:tasks.workunit.client.0.smithi047.stdout:leWrite 2024-03-30T07:26:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8472 ms) 2024-03-30T07:26:27.039 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8883 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7603 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7730 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (8460 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7069 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8109 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6754 ms) 2024-03-30T07:26:27.040 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7754 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (10018 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7903 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7834 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (25121 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (8058 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6684 ms) 2024-03-30T07:26:27.041 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7011 ms) 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (143463 ms total) 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [----------] Global test environment tear-down 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (516773 ms total) 2024-03-30T07:26:27.042 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=14 expected=14 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=3 expected=3 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-03-30T07:26:29.702 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=1 expected=1 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=2 expected=2 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:e960b815:::9:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=9 expected=9 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-03-30T07:26:29.703 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=12 expected=12 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=4 expected=4 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: : entry=6 expected=6 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (9955 ms) 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (374447 ms) 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-03-30T07:26:29.704 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (17390 ms) 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 7 tests from LibRadosList (507623 ms total) 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1943 ms) 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo1,foo2,foo3 2024-03-30T07:26:29.705 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo1 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo2 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo3 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo1,foo4,foo5 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo4 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo5 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo1 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset foo6,foo7 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo7 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: foo6 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-30T07:26:29.706 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo4 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo5 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns2:foo7 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns2:foo6 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: ns1:foo1 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo1 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo2 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: :foo3 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (1213 ms) 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 1 0 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 10 0 2024-03-30T07:26:29.707 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 13 0 2024-03-30T07:26:31.947 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:59.758 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:31.997 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:59.774 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:31.999 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:59.776 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.016 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:59.827 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.017 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:59.828 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.020 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:59.831 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.046 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:10:59.823 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.067 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:10:59.877 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:32.072 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:10:59.883 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 7 0 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 14 0 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 0 0 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 15 0 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 11 0 2024-03-30T07:26:37.167 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 5 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 8 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 6 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 3 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 4 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 12 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 9 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 2 0 2024-03-30T07:26:37.168 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (2083 ms) 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 3 tests from LibRadosListEC (5239 ms total) 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3232 ms) 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] 1 test from LibRadosListNP (3232 ms total) 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [----------] Global test environment tear-down 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [==========] 11 tests from 3 test suites ran. (526848 ms total) 2024-03-30T07:26:37.169 INFO:tasks.workunit.client.0.smithi047.stdout: api_list: [ PASSED ] 11 tests. 2024-03-30T07:26:52.164 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:19.973 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.194 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:20.005 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.215 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:19.992 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.217 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:19.994 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.222 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.032 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.223 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.034 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.238 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:20.049 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.259 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:20.036 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:26:52.276 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:20.086 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.404 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:31.181 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.406 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:31.183 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.406 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:31.217 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.417 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:31.227 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.449 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:31.226 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.474 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:31.284 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.491 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:31.301 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.493 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:31.304 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:03.547 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:31.357 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.166 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:34.976 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.169 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:34.979 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.187 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:34.997 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.221 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:35.031 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.224 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:35.001 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.224 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:35.035 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.226 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:35.003 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.243 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:35.054 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:07.244 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:35.021 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.318 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:40.094 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.320 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:40.097 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.328 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:40.139 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.332 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:40.143 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.346 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:40.123 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.374 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:40.184 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.386 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:40.196 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.390 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:40.201 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:12.419 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:40.230 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.191 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:50.001 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.244 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:50.054 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.275 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:11:50.085 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.292 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:50.069 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.293 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:50.104 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.295 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:50.072 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.296 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:50.107 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.323 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:11:50.133 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:22.323 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:11:50.100 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout:osAio.RoundTripAppendPP (4912 ms) 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3352 ms) 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (4539 ms) 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (84324 ms) 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-30T07:27:23.713 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: setting pool EIO 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: max_success 0, min_failed 1 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5244 ms) 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4901 ms) 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (379377 ms total) 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (6659 ms) 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9996 ms) 2024-03-30T07:27:23.714 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15361 ms) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3133 ms) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (35149 ms total) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2984 ms) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2984 ms total) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (16240 ms) 2024-03-30T07:27:23.715 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (60110 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (8221 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7675 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3102 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7289 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7991 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6945 ms) 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-30T07:27:23.716 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6902 ms) 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8204 ms) 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (8738 ms) 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7768 ms) 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-30T07:27:23.717 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6751 ms) 2024-03-30T07:27:32.288 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:00.099 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.294 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:00.104 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.322 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:00.132 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.341 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:00.152 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.345 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:00.155 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.346 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:00.122 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.347 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:00.124 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.372 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:00.149 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:32.373 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:00.183 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.377 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:05.187 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.444 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:05.255 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.449 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:05.260 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.467 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:05.277 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.511 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.287 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.515 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:05.325 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.519 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.296 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.539 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:05.316 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:37.541 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:05.351 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.612 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:20.422 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.618 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:20.428 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.669 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:20.446 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.673 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:20.450 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.674 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:20.485 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.689 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:20.499 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.693 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:20.504 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.712 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:20.488 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:27:52.735 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:20.546 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.498 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:35.308 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.503 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:35.313 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.538 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:35.349 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.552 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:35.329 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.553 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:35.330 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.554 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:35.365 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.557 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:35.367 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.587 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:35.364 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:07.602 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:35.413 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3224 ms) 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6877 ms) 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (8256 ms) 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7042 ms) 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-30T07:28:11.213 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7782 ms) 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6645 ms) 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7672 ms) 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (203434 ms total) 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (620944 ms total) 2024-03-30T07:28:11.214 INFO:tasks.workunit.client.0.smithi047.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-30T07:28:12.618 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.395 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.619 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.396 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.622 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:40.432 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.626 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:40.436 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.638 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:40.415 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.656 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:40.466 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.695 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:40.506 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.696 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:40.507 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:12.714 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:40.524 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.514 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:50.325 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.565 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:50.375 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.582 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:12:50.392 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.612 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:50.389 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.615 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:50.391 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.623 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:50.433 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.630 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:12:50.406 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.630 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:50.436 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:22.642 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:12:50.452 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.595 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:05.405 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.599 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:05.409 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.613 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:05.423 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.644 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:05.454 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.647 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:05.457 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.653 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.430 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.657 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.434 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.661 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:05.471 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:37.672 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:05.449 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.608 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:10.418 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.611 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:10.421 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.624 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:10.434 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.660 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:10.437 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.661 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:10.438 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.671 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:10.481 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.672 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:10.449 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.673 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:10.484 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:42.686 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:10.496 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.692 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:25.502 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.693 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:25.504 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.727 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.504 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.731 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.508 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.738 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:25.549 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.743 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:25.553 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.746 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:25.557 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.759 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:25.536 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:57.775 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:25.585 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.520 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:26.297 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.521 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:26.298 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.532 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:26.309 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.597 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:26.407 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.598 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:26.408 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.601 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:26.411 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.602 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:26.413 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.607 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:26.417 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:28:58.611 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:26.422 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.741 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.517 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.742 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.519 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.742 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:45.553 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.746 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:45.556 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.780 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:45.557 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.787 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:45.597 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.808 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:45.618 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.826 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:45.636 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:17.867 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:45.677 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.775 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:55.585 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.779 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:55.590 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.803 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:13:55.614 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.829 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:55.605 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.830 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:55.607 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.847 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:55.657 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.848 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:55.659 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.851 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:13:55.628 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:27.929 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:13:55.739 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.853 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:05.663 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.861 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:05.672 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.901 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:05.711 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.915 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.692 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.916 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.693 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.932 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:05.742 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.934 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:05.745 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.942 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:05.718 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:37.959 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:05.769 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.796 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.573 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.797 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.574 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.808 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:06.585 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.872 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:06.683 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.873 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:06.684 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.884 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:06.694 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.920 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:06.730 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.920 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:06.731 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:38.930 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:06.741 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.001 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:10.811 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.005 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:10.815 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.007 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:10.817 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.010 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:10.787 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.010 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:10.820 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.012 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:10.789 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.082 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:10.893 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.088 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:10.865 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:43.092 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:10.902 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:57.638 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-30T07:29:57.638 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-30T07:29:57.638 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-30T07:29:57.638 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: seed 40956 2024-03-30T07:29:57.638 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (18503 ms) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (113417 ms) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (16 ms) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (131936 ms total) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (25654 ms) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (28351 ms) 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-30T07:29:57.639 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (63362 ms) 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: my_snaps [4] 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: my_snaps [5,4] 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: my_snaps [6,5,4] 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: promoting some heads 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: promoting from clones for snap 7 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: done waiting 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (279685 ms) 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-30T07:29:57.640 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10236 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10139 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8803 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9372 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11700 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8929 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10966 ms) 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-30T07:29:57.641 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (42282 ms) 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10019 ms) 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (23325 ms) 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14894 ms) 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10113 ms) 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-30T07:29:57.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (10279 ms) 2024-03-30T07:29:58.135 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:25.945 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.139 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:25.950 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.204 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:25.981 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.204 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:26.014 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.208 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:25.985 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.209 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:26.019 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.210 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:26.020 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.237 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:26.013 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:29:58.244 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:26.054 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.137 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:40.947 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.140 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:40.950 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.170 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:40.980 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.193 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:41.003 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.193 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:40.970 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.194 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:40.971 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.195 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:41.005 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.227 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:41.038 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:13.233 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:41.010 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.027 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:45.837 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.032 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:45.843 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.045 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:14:45.856 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.081 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:45.891 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.081 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:45.858 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.084 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:45.861 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.087 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:45.897 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.095 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:14:45.871 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.097 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:14:45.907 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: ok, hit_set contains 201:602f83fe:::foo:head 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8857 ms) 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg_num = 32 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 0 ls 1711783813,0 2024-03-30T07:30:18.543 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 1 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 2 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 3 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 4 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 5 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 6 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 7 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 8 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 9 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 10 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 11 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 12 ls 1711783813,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 13 ls 1711783814,0 2024-03-30T07:30:18.544 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 14 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 15 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 16 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 17 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 18 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 19 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 20 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 21 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 22 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 23 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 24 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 25 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 26 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 27 ls 1711783814,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 28 ls 1711783813,0 2024-03-30T07:30:18.545 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 29 ls 1711783813,0 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 30 ls 1711783814,0 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg 31 ls 1711783813,0 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: pg_num = 32 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6cac518f:::0:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:02547ec2:::1:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f905c69b:::2:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:cfc208b3:::3:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d83876eb:::4:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b29083e3:::5:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c4fdafeb:::6:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:5c6b0b28:::7:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:bd63b0f1:::8:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e960b815:::9:head 2024-03-30T07:30:18.546 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:52ea6a34:::10:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:89d3ae78:::11:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:de5d7c5f:::12:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:566253c9:::13:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:62a1935d:::14:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:863748b0:::15:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3958e169:::16:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4d4dabf9:::17:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:8391935d:::18:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:28883081:::19:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:69259c59:::20:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4bdb80b7:::21:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a11c5d71:::22:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:271af37b:::23:head 2024-03-30T07:30:18.547 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:95b121be:::24:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:58d1031b:::25:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:0a050783:::26:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c709704c:::27:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:cbe56eaf:::28:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:86b4b162:::29:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:70d89383:::30:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:dd450c7c:::31:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6d5729b1:::32:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c388f3fb:::33:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:56cfea31:::34:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9dbc1bf7:::35:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:40b74ccd:::36:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4d5aaf42:::37:head 2024-03-30T07:30:18.548 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:920f362c:::38:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6cc53222:::39:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9cad833f:::40:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:1ea84d41:::41:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c4480ef6:::42:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a694361e:::43:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d1bd33e9:::44:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ddc2cd5d:::45:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:2b782207:::46:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7b187fca:::47:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:90ecdf6f:::48:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a5ed95fe:::49:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ea0eaa55:::50:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f33ef17b:::51:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a0d1b2f6:::52:head 2024-03-30T07:30:18.549 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:60c5229e:::53:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:edcbc575:::54:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:102cf253:::55:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:efb7fb0b:::56:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:50d0a326:::57:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d4dc5daf:::58:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3a130462:::59:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ec87ed71:::60:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d5bc9454:::61:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3ddfe313:::62:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7c2816b9:::63:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:47e00e4d:::64:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c6410c18:::65:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b48ed237:::66:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:cd63ad31:::67:head 2024-03-30T07:30:18.550 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b179e92b:::68:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:0d9f741a:::69:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6d3352ae:::70:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c6d5c19e:::71:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:bc4729c3:::72:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:77e930b9:::73:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:0abeecfd:::74:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b7c37e15:::75:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b6378398:::76:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:02bd68de:::77:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:cc795d2d:::78:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:630d4fea:::79:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e0d29ef5:::80:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:fd6f13d2:::81:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:606461d5:::82:head 2024-03-30T07:30:18.551 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:eadbdc43:::83:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:8761d0bb:::84:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9ef0186f:::85:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e0d41294:::86:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:961de695:::87:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:1423148f:::88:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:633a8fa2:::89:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a8653809:::90:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3dac8b33:::91:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:35aad435:::92:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f6dcc343:::93:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:dbbdad87:::94:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:1cb48ce0:::95:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:03cd461c:::96:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:17a4ea99:::97:head 2024-03-30T07:30:18.552 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9993c9a7:::98:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6394211c:::99:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:94c7ae57:::100:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6fdee5bb:::101:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9a477fd1:::102:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:eb850916:::103:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:affc56b9:::104:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b42dc814:::105:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f319f8f0:::106:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9a40b9de:::107:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:8b524f28:::108:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e3de589f:::109:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:90f90a5b:::110:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a7b4f1d7:::111:head 2024-03-30T07:30:18.553 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:af51766e:::112:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b6f90bd1:::113:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e0261208:::114:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c9569ef7:::115:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:61bebe50:::116:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:fe93412b:::117:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d3d38bee:::118:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3100ba0c:::119:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d0560ada:::120:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f0ea8b35:::121:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:766f231a:::122:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a07a2582:::123:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:bd7c6b3a:::124:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:fb2ddaff:::125:head 2024-03-30T07:30:18.554 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4408e1fe:::126:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ee1df7a7:::127:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c3002909:::128:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4f48ffa9:::129:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:edf38733:::130:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c08425c0:::131:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:5f902d98:::132:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:41ea2c93:::133:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:813cee13:::134:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:0131818d:::135:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:26ba5a85:::136:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:381b8a5a:::137:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:28797e47:::138:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:bfca7f22:::139:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:36807075:::140:head 2024-03-30T07:30:18.555 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:80b03975:::141:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:5c15709b:::142:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f39ea15e:::143:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ea992956:::144:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:48887b1c:::145:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:9f24a9dd:::146:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:987f100b:::147:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d2dd3581:::148:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7fed1808:::149:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c80b70e9:::150:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:85ed90f9:::151:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:36428b24:::152:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d044c34a:::153:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7c18bf58:::154:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d1c21232:::155:head 2024-03-30T07:30:18.556 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a7a3c575:::156:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:87da0633:::157:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d5ac3822:::158:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3f20522d:::159:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6ca26563:::160:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:532ce135:::161:head 2024-03-30T07:30:18.557 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:c78863e6:::162:head 2024-03-30T07:30:33.192 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:01.003 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.197 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:01.007 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.220 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:01.030 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.244 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:01.021 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.245 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:01.022 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.259 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:01.069 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.261 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:01.071 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.282 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:01.059 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:33.288 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:01.098 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.089 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:05.899 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.090 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:05.901 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.102 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:05.912 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.135 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:05.946 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.140 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:05.950 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.140 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:05.917 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.141 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:05.918 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.154 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:05.931 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:38.155 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:05.965 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.309 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:21.119 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.317 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:21.127 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.344 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:21.154 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.390 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:21.167 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.396 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:21.173 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.402 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:21.212 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.404 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:21.214 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.419 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:21.196 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:53.434 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:21.244 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.289 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:26.099 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.291 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:26.101 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.310 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:26.120 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.341 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:26.151 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.342 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:26.152 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.342 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.119 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.344 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.121 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.357 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:26.167 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:30:58.358 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:26.135 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.370 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:36.180 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.375 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:36.186 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.411 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:36.221 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.435 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:36.245 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.438 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:36.215 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.441 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:36.251 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.447 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:36.223 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.472 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:36.283 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:08.478 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:36.255 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.375 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:37.186 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.381 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:37.191 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.398 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:37.174 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.398 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:37.209 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.401 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:37.212 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.417 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:37.194 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.476 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:37.286 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.481 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:37.291 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:09.502 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:37.279 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.376 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:46.186 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.378 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:46.188 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.394 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:46.204 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.426 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:46.237 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.427 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:46.238 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.431 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:46.207 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.432 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:46.209 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.442 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:46.253 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:18.448 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:46.225 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:27.303 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7462ddf6:::163:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:5d165639:::164:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f43765fc:::165:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b4c720e9:::166:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e694b040:::167:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:afa38db2:::168:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:77ba9f53:::169:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:87495034:::170:head 2024-03-30T07:31:27.304 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7c96bf0e:::171:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:dbe346cc:::172:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:e943ec24:::173:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f97a9c0c:::174:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:6f26e74d:::175:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4f95e106:::176:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:0e6f2f8f:::177:head 2024-03-30T07:31:27.305 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:05db05f1:::178:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:38a78d66:::179:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:d095610b:::180:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:a1a9d709:::181:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:1e5d39db:::182:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:f7df4fb9:::183:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:03a7f161:::184:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ba70721e:::185:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:28e5662d:::186:head 2024-03-30T07:31:27.306 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:973d52de:::187:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:4303eb1c:::188:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:b990b48e:::189:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:29b8165b:::190:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:3547f197:::191:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:7e260936:::192:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:1abec7b1:::193:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:10fdda93:::194:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:15817eea:::195:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:770bab57:::196:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:ed9e13e7:::197:head 2024-03-30T07:31:27.307 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:71471a8f:::198:head 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: checking for 202:10fb1d02:::199:head 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (21673 ms) 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,0 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: first is 1711783836 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,0 2024-03-30T07:31:27.308 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,1711783844,1711783845,0 2024-03-30T07:31:27.309 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,1711783844,1711783845,0 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783836,1711783838,1711783839,1711783841,1711783842,1711783844,1711783845,0 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711783839,1711783841,1711783842,1711783844,1711783845,1711783847,1711783848,0 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: first now 1711783839, trimmed 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22442 ms) 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: foo0 2024-03-30T07:31:27.310 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T07:31:27.311 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16514 ms) 2024-03-30T07:31:27.311 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-30T07:31:28.437 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:56.247 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.442 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:56.253 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.478 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:15:56.288 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.503 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:56.280 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.511 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:56.288 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.522 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:56.333 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.527 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:56.337 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.534 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:15:56.310 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:28.555 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:15:56.365 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.460 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:06.269 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.461 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:06.271 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.493 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:06.304 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.506 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:06.283 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.509 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:06.286 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.519 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:06.329 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.521 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:06.331 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.531 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:06.308 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:38.548 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:06.358 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.618 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:11.428 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.622 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:11.432 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.635 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:11.445 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.671 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.447 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.672 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:11.482 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.672 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.449 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.673 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:11.484 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.684 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:11.461 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:43.686 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:11.496 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.547 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:26.357 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.551 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:26.362 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.580 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:26.390 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.601 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:26.377 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.603 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:26.380 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.605 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:26.415 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.608 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:26.418 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.627 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:26.403 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:58.631 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:26.441 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.641 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:27.451 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.642 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:27.452 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.652 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:27.462 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.653 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:27.429 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.653 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:27.430 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.665 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:27.442 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.833 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:27.643 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.834 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:27.644 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:31:59.844 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:27.654 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.661 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:41.471 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.666 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:41.476 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.698 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:41.508 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.712 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:41.489 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.715 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:41.525 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.716 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:41.493 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.717 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:41.527 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.744 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:41.521 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:13.747 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:41.557 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.738 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:46.548 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.742 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:46.552 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.759 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:46.569 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.789 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.565 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.794 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:46.604 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.796 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:46.606 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.806 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.583 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.811 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:46.621 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:18.822 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:46.598 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.763 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:56.573 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.813 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:56.589 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.814 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:56.591 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.821 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:56.632 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.835 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:16:56.612 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.844 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:16:56.655 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.874 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:56.684 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.876 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:56.686 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:28.894 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:16:56.704 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.759 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:06.568 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.764 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:06.574 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.780 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:06.590 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.812 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:06.622 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.813 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:06.590 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.816 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:06.593 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.816 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:06.626 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.830 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:06.607 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:38.833 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:06.643 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.812 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:16.621 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.816 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:16.626 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.832 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:16.642 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.861 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:16.672 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.867 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:16.677 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.869 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:16.646 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.871 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:16.647 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.884 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:16.661 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:48.886 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:16.696 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.095 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:26.904 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.097 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:26.907 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.121 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:26.931 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.163 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:26.973 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.164 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:26.940 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.167 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:26.977 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.172 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:26.949 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.196 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:27.006 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:32:59.209 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:26.985 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.828 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:31.638 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.878 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:31.688 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.895 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:31.705 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.925 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.701 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.929 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.706 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.934 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:31.744 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.936 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:31.746 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.948 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:31.725 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:03.951 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:31.761 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.035 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:46.844 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.038 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:46.848 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.062 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:46.871 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.098 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:46.907 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.099 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:46.909 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.104 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:46.881 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.107 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:46.884 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.127 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:46.936 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:19.132 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:46.908 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.245 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:53.055 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.300 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:53.110 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.322 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:53.132 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.356 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:53.132 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.357 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:53.134 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.362 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:53.170 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.362 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:53.171 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.382 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:53.192 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:25.388 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:53.165 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:28.981 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:56.791 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:28.984 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:56.794 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:28.999 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:17:56.809 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.035 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:56.845 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.037 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:56.847 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.038 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:56.814 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.041 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:56.817 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.054 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:17:56.831 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:29.054 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:17:56.864 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.046 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:11.856 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.048 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:11.858 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.065 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:11.875 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.098 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:11.907 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.099 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:11.909 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.104 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:11.881 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.105 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:11.882 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.118 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:11.928 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:44.123 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:11.899 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.062 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:16.872 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.065 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:16.875 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.076 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:16.886 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.116 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:16.926 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.117 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:16.927 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.118 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:16.894 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.119 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:16.896 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.128 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:16.938 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:33:49.129 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:16.906 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.112 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:31.922 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.114 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:31.924 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.147 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:31.957 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.166 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:31.943 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.169 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:31.945 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.174 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:31.984 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.176 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:31.986 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.191 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:31.967 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.199 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:32.009 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.922 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.698 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.922 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.699 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.930 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:32.707 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.950 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:32.760 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.951 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:32.761 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.963 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:32.773 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.977 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:32.787 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.978 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:32.788 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:04.986 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:32.796 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:13.145 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:34:13.132+0000 26691640 -1 snap_mapper.add_oid found existing snaps mapped on 77:3cc8cb25:test-rados-api-smithi047-40956-76::foo:28, removing 2024-03-30T07:34:13.160 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:34:13.150+0000 26e92640 -1 snap_mapper.add_oid found existing snaps mapped on 77:3cc8cb25:test-rados-api-smithi047-40956-76::foo:28, removing 2024-03-30T07:34:19.133 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:46.943 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.190 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:47.000 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.221 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:47.031 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.237 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:47.014 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.240 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:47.017 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.242 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:47.051 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.242 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:34:19.225+0000 26291640 -1 snap_mapper.add_oid found existing snaps mapped on 77:55264efe:test-rados-api-smithi047-40956-78::foo:29, removing 2024-03-30T07:34:19.247 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:47.057 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.253 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:34:19.243+0000 25a90640 -1 snap_mapper.add_oid found existing snaps mapped on 77:55264efe:test-rados-api-smithi047-40956-78::foo:29, removing 2024-03-30T07:34:19.266 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:47.042 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:19.279 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:47.089 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.199 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:52.009 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.203 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:52.013 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.221 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:18:52.030 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.249 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:52.059 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.253 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:52.063 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.267 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.043 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.270 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.046 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.273 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:18:52.083 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:24.283 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:18:52.060 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.234 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:02.043 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.237 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:02.047 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.254 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:02.064 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.287 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:02.063 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.287 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:02.097 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.289 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:02.066 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.291 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:02.101 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.304 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:02.081 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:34.310 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:02.120 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.261 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:12.071 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.263 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:12.073 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.280 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:12.090 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.313 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.089 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.314 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:12.124 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.318 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.094 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.318 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:12.128 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.332 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:12.109 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:44.335 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:12.145 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:45.921 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:34:45.904+0000 22289640 -1 snap_mapper.add_oid found existing snaps mapped on 220:42385dde:test-rados-api-smithi047-40956-80::foo:2, removing 2024-03-30T07:34:45.938 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:34:45.926+0000 29e98640 -1 snap_mapper.add_oid found existing snaps mapped on 220:42385dde:test-rados-api-smithi047-40956-80::foo:2, removing 2024-03-30T07:34:54.327 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:22.137 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.329 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:22.139 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.351 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:22.161 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.374 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:22.184 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.378 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.154 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.379 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.156 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.380 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:22.190 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.400 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:22.176 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:34:54.403 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:22.213 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.350 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:32.160 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.352 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:32.162 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.379 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:32.189 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.403 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:32.213 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.404 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.180 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.405 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.182 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.406 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:32.216 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.431 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:32.241 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:04.436 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:32.212 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.365 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:37.175 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.367 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:37.177 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.382 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:37.192 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.414 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:37.190 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.417 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:37.193 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.422 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:37.232 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.425 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:37.235 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.428 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:37.205 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:09.437 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:37.247 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.711 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:52.520 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.713 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:52.523 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.765 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:52.575 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.796 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:52.573 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.803 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:52.580 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.804 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:52.615 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.808 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:52.618 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.843 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:52.619 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:24.850 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:52.659 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.700 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:53.510 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.701 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:53.477 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.701 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:53.511 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.702 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:53.479 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.713 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:19:53.523 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.714 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:19:53.491 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.736 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:53.546 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.737 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:53.547 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:25.747 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:19:53.557 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20179 ms) 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25651 ms) 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3741 ms) 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5410 ms) 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4306 ms) 2024-03-30T07:35:26.642 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4549 ms) 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5157 ms) 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (41087 ms) 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18095 ms) 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5422 ms) 2024-03-30T07:35:26.643 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4499 ms) 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27292 ms) 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16100 ms) 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6609 ms) 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5582 ms) 2024-03-30T07:35:26.644 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: waiting for scrubs... 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: done waiting 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25767 ms) 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10633 ms) 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (16245 ms) 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10665 ms) 2024-03-30T07:35:26.645 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-30T07:35:39.501 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:07.311 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.504 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:07.314 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.512 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:07.322 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.514 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:07.324 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.521 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:07.297 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.522 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:07.299 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.590 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:07.400 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.597 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:07.408 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:39.611 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:07.388 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.293 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:08.070 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.294 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:08.071 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.305 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:08.081 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.436 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:08.246 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.438 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:08.248 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.438 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:08.248 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.439 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:08.249 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.450 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:08.260 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:40.452 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:08.262 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:44.996 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:12.806 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.000 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:12.810 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.017 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:12.827 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.057 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:12.867 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.061 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.838 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.064 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:12.874 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.072 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.848 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.083 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:12.893 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:45.099 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:12.876 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.867 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:27.677 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.871 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:27.682 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.898 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:27.708 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.922 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.698 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.925 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.702 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.934 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:27.744 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.942 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:27.752 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.952 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:27.728 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:35:59.969 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:27.779 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:04.987 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:32.797 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:04.998 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:32.808 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.015 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:32.825 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.051 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:32.827 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.052 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:32.829 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.065 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:32.842 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.069 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:32.879 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.070 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:32.881 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:05.084 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:32.894 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.059 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:47.869 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.063 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:47.874 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.095 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:47.905 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.144 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.921 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.146 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.923 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.148 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:47.958 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.151 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:47.962 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.167 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:47.944 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:20.173 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:47.983 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:29.989 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:57.798 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:29.993 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:57.803 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.008 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:20:57.818 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.044 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.820 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.047 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.823 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.047 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:57.857 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.049 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:57.859 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.060 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:20:57.836 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.062 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:20:57.872 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:30.417 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:36:30.378+0000 22689640 -1 snap_mapper.add_oid found existing snaps mapped on 77:9f570c86:test-rados-api-smithi047-40956-97::foo:31, removing 2024-03-30T07:36:30.424 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:36:30.415+0000 29697640 -1 snap_mapper.add_oid found existing snaps mapped on 77:9f570c86:test-rados-api-smithi047-40956-97::foo:31, removing 2024-03-30T07:36:34.980 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:02.790 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:34.984 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:02.794 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.000 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:02.810 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.030 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:02.840 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.033 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.809 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.033 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:02.843 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.034 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.810 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.047 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:02.824 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:35.053 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:02.863 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.185 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:17.994 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.187 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:17.998 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.232 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:18.043 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.236 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.013 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.238 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.014 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.253 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:18.063 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.261 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:18.071 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.267 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:18.044 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:36:50.295 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:18.105 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.116 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:32.926 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.119 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:32.929 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.146 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:32.957 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.166 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:32.976 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.168 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.945 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.169 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.946 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.172 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:32.982 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.196 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:32.973 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:05.201 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:33.011 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.117 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:37.927 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.122 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:37.932 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.136 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:37.947 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.176 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:37.986 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.177 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:37.953 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.179 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:37.956 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.180 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:37.990 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.190 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:37.967 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:10.193 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:38.003 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.146 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:47.956 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.147 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:47.958 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.170 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:47.980 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.199 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:47.976 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.200 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:47.977 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.211 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:48.022 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.213 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:48.023 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.221 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:47.997 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:20.236 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:48.046 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.133 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:57.942 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.180 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:57.957 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.182 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:57.993 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.183 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:57.960 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.218 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:21:58.029 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.222 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:21:57.999 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.250 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:58.060 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.256 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:58.067 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:30.293 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:21:58.103 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.274 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:08.084 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.325 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:08.101 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.325 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:08.102 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.332 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:08.143 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.334 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:08.145 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.343 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:08.153 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.351 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:08.128 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.363 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:08.173 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:40.395 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:08.206 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.461 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:13.271 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.511 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:13.321 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.517 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:13.293 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.520 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:13.297 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.530 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:13.340 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.537 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:13.314 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.565 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:13.376 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.566 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:13.377 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:45.584 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:13.394 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.260 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:18.070 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.266 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:18.077 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.284 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:18.094 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.324 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.101 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.325 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.102 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.328 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:18.139 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.330 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:18.141 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.335 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:18.112 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:37:50.345 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:18.156 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.291 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:28.101 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.296 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:28.107 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.316 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:28.127 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.348 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:28.158 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.352 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:28.129 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.353 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:28.164 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.354 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:28.130 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.368 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:28.145 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:00.370 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:28.181 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.321 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:38.131 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.325 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:38.136 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.351 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:38.161 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.373 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.149 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.377 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:38.187 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.377 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.154 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.382 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:38.193 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.402 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:38.179 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:10.408 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:38.219 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.296 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:43.106 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.345 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:43.122 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.346 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:43.123 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.352 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:43.162 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.362 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:43.139 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.373 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:43.183 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.396 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:43.207 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.406 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:43.216 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:15.423 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:43.233 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.415 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:58.225 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.418 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:58.228 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.452 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:58.262 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.470 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.247 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.473 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.250 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.487 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:58.298 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.490 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:58.300 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.504 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.281 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:30.520 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:58.330 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.171 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.947 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.172 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.949 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.180 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:22:58.957 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.248 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:59.058 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.249 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:59.059 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.259 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:22:59.070 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.294 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:59.104 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.295 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:59.106 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:31.304 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:22:59.115 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.531 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:18.341 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.593 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:18.403 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.618 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:18.429 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.658 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.435 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.659 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.436 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.666 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:18.477 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.672 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:18.483 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.682 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:18.459 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:38:50.694 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:18.504 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.555 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:33.365 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.558 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:33.368 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.592 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:33.402 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.608 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.385 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.609 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:33.419 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.609 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.386 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.613 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:33.423 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.638 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:33.415 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:05.654 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:33.464 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.757 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:38.568 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.804 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.581 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.806 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.582 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.817 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:38.628 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.823 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:38.600 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.842 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:38.653 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.869 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:38.680 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.870 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:38.681 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:10.886 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:38.696 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.685 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:48.495 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.687 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:48.497 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.713 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:23:48.523 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.736 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.513 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.738 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.515 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.752 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:48.562 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.754 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:48.565 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.759 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:23:48.536 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:20.776 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:23:48.586 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.692 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:03.502 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.744 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:03.554 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.773 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:03.584 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.791 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:03.568 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.795 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:03.572 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.812 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:03.622 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.816 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:03.627 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.823 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:03.600 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:35.870 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:03.680 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.537 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:04.314 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.538 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:04.315 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.546 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:04.323 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.631 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:04.441 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.632 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:04.442 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.641 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:04.451 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.664 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:04.475 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.665 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:04.476 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:36.678 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:04.488 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10925 ms) 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8283 ms) 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-30T07:39:49.047 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5859 ms) 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26988 ms) 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.048 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15288 ms) 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (14008 ms) 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 00000012 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12785 ms) 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: require_osd_release = reef 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (14411 ms) 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (1013132 ms total) 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-30T07:39:49.049 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1513 ms) 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10807 ms) 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (34451 ms) 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (19 ms) 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (46790 ms total) 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-30T07:39:49.050 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7852 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9093 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: waiting for scrub... 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: done waiting 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (28012 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11445 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10935 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10056 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12059 ms) 2024-03-30T07:39:49.051 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-30T07:39:51.035 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:18.845 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.187 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:18.849 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.188 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:18.885 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.188 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.882 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.188 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.890 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.188 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:18.930 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.188 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:18.898 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.189 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:18.930 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.189 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:18.981 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.946 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:19.723 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.947 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:19.724 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:51.964 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:19.741 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.193 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:20.003 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.194 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:20.005 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.217 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:20.028 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.269 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:20.079 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.270 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:20.080 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:52.285 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:20.095 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:55.958 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:23.768 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.012 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:23.823 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.021 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.799 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.026 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.803 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.027 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:23.838 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.040 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:23.817 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.067 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:23.878 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.069 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:23.879 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:39:56.091 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:23.893 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.214 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:34.024 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.220 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:34.031 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.242 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:34.053 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.293 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.070 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.294 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.071 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.313 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:34.090 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.320 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:34.131 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.325 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:34.136 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:06.345 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:34.156 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:15.929 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:43.739 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:15.936 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:43.746 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:15.963 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:43.773 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:15.980 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:43.757 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:15.985 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:43.762 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:16.000 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:43.811 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:16.006 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:43.784 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:16.017 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:43.827 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:16.041 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:43.851 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:20.934 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:48.744 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:20.937 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:48.748 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:20.955 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:24:48.765 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:20.988 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:48.799 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:20.990 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:48.800 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:21.004 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:48.781 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:21.006 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:48.783 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:21.011 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:24:48.821 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:21.020 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:24:48.797 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.099 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:03.909 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.102 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:03.912 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.146 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:03.923 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.148 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:03.958 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.149 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:03.927 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.170 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:03.980 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.176 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:03.987 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.179 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:03.956 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:36.208 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:04.018 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.040 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:18.849 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.040 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:18.851 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.063 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:18.874 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.081 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:18.858 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.083 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:18.861 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.095 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:18.905 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.099 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:18.909 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.105 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:18.882 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:51.122 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:18.932 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.058 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:23.868 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.061 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:23.871 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.073 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:23.884 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.111 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:23.921 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.111 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:23.888 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.112 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:23.923 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.113 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:23.890 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.124 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:23.901 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:40:56.125 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:23.935 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.044 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:33.854 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.098 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:33.909 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.128 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:33.939 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.147 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:33.957 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.147 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:33.924 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.149 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:33.926 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.151 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:33.962 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.173 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:33.950 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:06.180 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:33.991 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.293 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:44.103 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.353 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:44.130 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.356 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:44.133 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.359 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:44.170 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.362 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:44.173 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.366 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:44.177 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.382 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:44.159 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.390 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:44.201 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:16.451 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:44.262 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.147 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:48.958 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.152 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:48.963 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.169 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:25:48.979 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.199 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:48.976 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.202 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:49.012 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.202 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:48.979 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.206 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:49.017 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.218 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:25:48.995 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:21.227 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:25:49.037 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.281 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:04.092 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.333 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:04.144 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.354 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:04.164 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.377 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:04.188 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.378 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:04.155 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.379 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:04.156 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.384 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:04.194 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.396 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:04.173 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:36.407 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:04.218 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.260 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:19.070 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.266 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:19.077 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.302 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:19.112 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.312 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:19.123 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.322 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:19.133 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.323 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.099 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.324 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.101 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.355 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:19.165 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:51.356 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:19.133 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.248 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:20.025 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.249 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:20.027 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.263 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:20.040 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.317 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:20.128 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.318 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:20.129 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.327 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:20.137 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.328 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:20.139 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.333 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:20.144 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:52.368 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:20.179 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.473 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:24.283 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.475 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:24.252 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.476 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:24.287 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.477 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:24.255 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.484 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:24.295 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.486 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:24.296 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.560 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:24.337 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.576 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:24.386 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:41:56.584 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:24.394 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.320 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:34.130 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.322 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:34.132 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.340 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:34.150 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.371 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:34.181 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.372 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:34.183 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.375 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:34.152 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.377 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:34.154 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.393 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:34.204 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:06.396 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:34.173 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.338 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:44.148 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.340 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:44.151 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.362 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:44.173 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.388 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:44.165 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.390 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:44.167 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.406 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:44.217 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.408 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:44.185 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.410 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:44.221 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:16.429 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:44.239 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.353 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:49.163 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.356 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:49.166 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.380 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:26:49.190 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.401 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:49.178 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.404 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:49.181 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.409 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:49.220 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.410 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:49.221 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.418 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:26:49.196 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:21.425 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:26:49.236 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.440 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:04.250 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.444 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:04.255 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.474 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:04.284 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.491 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:04.268 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.492 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:04.303 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.494 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:04.271 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.499 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:04.309 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.516 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:04.293 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:36.522 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:04.333 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.239 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:05.016 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.239 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:05.017 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.248 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:05.025 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.262 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:05.073 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.263 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:05.074 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.275 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:05.086 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.291 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:05.102 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.292 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:05.103 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:37.301 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:05.112 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.541 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:19.351 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.605 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:19.416 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.631 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:19.441 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.656 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:19.433 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.657 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:19.435 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.661 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:19.472 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.664 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:19.475 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.683 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:19.461 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:51.690 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:19.500 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.494 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:24.304 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.497 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:24.308 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.511 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:24.321 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.541 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:24.318 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.543 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:24.320 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.548 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:24.358 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.552 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:24.363 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.556 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:24.333 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:42:56.566 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:24.377 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.534 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:39.345 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.536 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:39.347 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.552 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:39.363 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.582 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:39.358 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.584 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:39.394 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.586 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:39.363 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.592 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:39.403 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.606 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:39.383 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:11.614 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:39.424 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.556 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:49.367 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.562 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:49.372 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.577 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:49.387 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.607 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:49.383 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.610 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:49.387 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.611 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:49.422 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.616 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:49.427 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.625 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:49.402 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:21.636 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:49.447 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.587 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:59.397 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.589 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:59.400 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.603 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:27:59.413 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.637 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:59.414 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.640 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:59.450 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.640 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:59.417 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.641 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:59.451 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.655 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:27:59.432 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:31.657 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:27:59.467 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.607 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:09.417 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.611 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:09.422 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.625 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:09.436 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.658 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:09.435 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.659 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:09.437 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.660 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:09.471 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.663 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:09.474 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.674 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:09.451 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:41.680 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:09.490 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.703 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:19.513 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.708 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:19.519 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.722 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:19.533 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.752 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:19.528 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.753 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:19.530 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.758 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:19.569 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.759 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:19.570 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.766 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:19.543 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:43:51.771 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:19.582 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.732 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:29.543 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.735 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:29.546 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.748 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:29.558 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.775 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:29.552 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.777 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:29.555 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.782 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:29.592 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.785 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:29.596 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.791 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:29.568 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:01.803 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:29.614 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:02.400 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:44:02.397+0000 12a60640 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.753389+0000 front 2024-03-30T07:42:38.756865+0000 (oldest deadline 2024-03-30T07:44:01.645510+0000) 2024-03-30T07:44:02.673 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:44:02.670+0000 12a60640 -1 osd.0 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.206805+0000 front 2024-03-30T07:42:38.196916+0000 (oldest deadline 2024-03-30T07:44:01.727370+0000) 2024-03-30T07:44:03.014 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:44:03.011+0000 12a60640 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.600989+0000 front 2024-03-30T07:42:38.606557+0000 (oldest deadline 2024-03-30T07:44:02.129782+0000) 2024-03-30T07:44:03.198 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:44:03.192+0000 ea58640 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.354375+0000 front 2024-03-30T07:42:38.351867+0000 (oldest deadline 2024-03-30T07:44:02.442872+0000) 2024-03-30T07:44:03.310 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:44:03.300+0000 ea58640 -1 osd.1 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.817056+0000 front 2024-03-30T07:42:38.807498+0000 (oldest deadline 2024-03-30T07:44:02.903896+0000) 2024-03-30T07:44:03.408 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:44:03.404+0000 12a60640 -1 osd.4 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.753389+0000 front 2024-03-30T07:42:38.756865+0000 (oldest deadline 2024-03-30T07:44:01.645510+0000) 2024-03-30T07:44:03.666 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:44:03.664+0000 12a60640 -1 osd.2 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:40.584351+0000 front 2024-03-30T07:42:40.586497+0000 (oldest deadline 2024-03-30T07:44:02.874298+0000) 2024-03-30T07:44:03.667 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:44:03.665+0000 12a60640 -1 osd.0 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.206805+0000 front 2024-03-30T07:42:38.196916+0000 (oldest deadline 2024-03-30T07:44:01.727370+0000) 2024-03-30T07:44:04.017 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:44:04.013+0000 12a60640 -1 osd.7 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.600989+0000 front 2024-03-30T07:42:38.606557+0000 (oldest deadline 2024-03-30T07:44:02.129782+0000) 2024-03-30T07:44:04.207 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:44:04.202+0000 ea58640 -1 osd.3 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:40.503760+0000 front 2024-03-30T07:42:40.507990+0000 (oldest deadline 2024-03-30T07:44:03.995636+0000) 2024-03-30T07:44:04.243 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:44:04.234+0000 ea58640 -1 osd.5 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.354375+0000 front 2024-03-30T07:42:38.351867+0000 (oldest deadline 2024-03-30T07:44:02.442872+0000) 2024-03-30T07:44:04.329 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:44:04.324+0000 ea58640 -1 osd.1 970 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:42:38.817056+0000 front 2024-03-30T07:42:38.807498+0000 (oldest deadline 2024-03-30T07:44:02.903896+0000) 2024-03-30T07:44:11.755 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:39.565 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.760 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:39.571 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.782 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:39.592 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.805 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:39.582 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.810 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:39.587 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.830 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:39.640 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.831 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:39.608 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.831 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:39.642 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:11.852 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:39.663 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:16.321 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:44:16.317+0000 1025b640 -1 osd.6 977 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T07:44:21.886 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:49.696 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.890 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:49.701 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.910 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:49.721 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.937 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:49.714 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.939 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:49.716 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.958 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:49.735 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.982 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:49.793 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:21.985 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:49.796 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:22.006 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:49.816 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.618 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:54.428 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.682 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:54.492 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.714 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:54.524 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.753 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:54.530 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.757 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:54.534 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.759 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:54.569 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.765 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:54.575 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.785 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:54.561 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:26.810 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:54.620 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.625 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:55.402 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.626 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:55.403 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.638 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:28:55.415 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.969 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:55.780 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.971 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:55.781 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.982 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:28:55.792 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.986 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:55.797 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.987 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:55.798 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:27.998 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:28:55.808 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.011 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:09.822 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.063 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:09.873 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.086 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:09.897 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.113 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:09.890 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.116 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:09.927 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.117 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:09.894 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.124 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:09.934 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.135 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:09.912 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:42.140 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:09.950 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.195 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:15.006 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.198 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:15.009 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.218 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:15.028 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.257 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:15.034 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.259 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:15.036 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.274 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:15.052 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.276 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:15.087 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.279 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:15.090 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:47.301 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:15.112 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:49.274 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9839 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14338 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9322 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15281 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (9705 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (20736 ms) 2024-03-30T07:44:49.275 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: ok, hit_set contains 252:602f83fe:::foo:head 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (16150 ms) 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: first is 1711784482 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,0 2024-03-30T07:44:49.276 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,1711784491,1711784492,0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,1711784491,1711784492,0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784482,1711784484,1711784485,1711784487,1711784488,1711784491,1711784492,0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: got ls 1711784485,1711784487,1711784488,1711784491,1711784492,1711784494,1711784495,0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: first now 1711784485, trimmed 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22813 ms) 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: foo0 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (16766 ms) 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20518 ms) 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (26398 ms) 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-30T07:44:49.277 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (85714 ms) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (17704 ms) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (19028 ms) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3351 ms) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (397117 ms total) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1618876 ms total) 2024-03-30T07:44:49.278 INFO:tasks.workunit.client.0.smithi047.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-30T07:44:49.283 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40669 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40669 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40874 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40874 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41389 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41389 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41115 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41115 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41009 2024-03-30T07:44:49.284 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41009 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40741 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40741 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41065 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41065 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40780 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40780 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41263 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41263 2024-03-30T07:44:49.285 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40644 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40644 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40705 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40705 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41298 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41298 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40634 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40634 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40655 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40655 2024-03-30T07:44:49.286 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40930 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40930 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41038 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41038 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40992 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40992 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41359 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41359 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41229 2024-03-30T07:44:49.287 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41229 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40849 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40849 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41157 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41157 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40627 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40627 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40685 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40685 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41089 2024-03-30T07:44:49.288 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41089 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=40726 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 40726 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41200 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41200 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ for t in "${!pids[@]}" 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ pid=41212 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ wait 41212 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ exit 0 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ cleanup 2024-03-30T07:44:49.289 INFO:tasks.workunit.client.0.smithi047.stderr:+ pkill -P 40622 2024-03-30T07:44:49.295 INFO:tasks.workunit.client.0.smithi047.stderr:+ true 2024-03-30T07:44:49.295 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T07:44:49.295 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T07:44:49.363 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-03-30T07:44:49.364 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T07:44:49.693 DEBUG:teuthology.parallel:result is None 2024-03-30T07:44:49.694 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T07:44:49.720 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T07:44:49.720 DEBUG:teuthology.orchestra.run.smithi047:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T07:44:49.788 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T07:44:49.789 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-30T07:44:49.801 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T07:44:49.801 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 10 monclient: get_monmap_and_config 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 10 monclient: build_initial_monmap 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 10 monclient: monmap: 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:49.929749+0000 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:49.929749+0000 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:49.931 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:49.932 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:49.932 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:49.932 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:49.932 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 1 Processor -- start 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 1 -- start start 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.929+0000 7f4a70b18640 10 monclient: init 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4a6c0adad0 0x7f4a6c13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-a con 0x7f4a6c0adad0 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a6c0adeb0 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6c06e550 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:49.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a6c13e310 con 0x7f4a6c0adeb0 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a6c06eb90 con 0x7f4a6c0adad0 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c070a10 con 0x7f4a6c06e550 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.930+0000 7f4a70b18640 10 monclient(hunting): authenticate will time out at 3216.045758s 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a69d9b640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f4a6c0adeb0 0x7f4a6c071110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:33602/0 (socket says 172.21.15.47:33602) 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a69d9b640 1 -- 172.21.15.47:0/1300451772 learned_addr learned my addr 172.21.15.47:0/1300451772 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a6ad9d640 1 --2- 172.21.15.47:0/1300451772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a6ad9d640 10 monclient(hunting): get_auth_request con 0x7f4a6c06e550 auth_method 0 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a6ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a6ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.931+0000 7f4a6ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.932+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:49.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.932+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:49.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.932+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:49.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.934+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_done global_id 17051 payload 274 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.934+0000 7f4a6ad9d640 1 -- 172.21.15.47:0/1300451772 >> v2:172.21.15.47:3300/0 conn(0x7f4a6c0adad0 msgr2=0x7f4a6c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.934+0000 7f4a6ad9d640 1 --2- 172.21.15.47:0/1300451772 >> v2:172.21.15.47:3300/0 conn(0x7f4a6c0adad0 0x7f4a6c13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.934+0000 7f4a6ad9d640 1 -- 172.21.15.47:0/1300451772 >> v1:172.21.15.52:6789/0 conn(0x7f4a6c0adeb0 legacy=0x7f4a6c071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.935+0000 7f4a6ad9d640 10 monclient: _finish_hunting 0 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.935+0000 7f4a6ad9d640 1 monclient: found mon.noname-b 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.935+0000 7f4a6ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.935+0000 7f4a6ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:49.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.935+0000 7f4a6ad9d640 1 -- 172.21.15.47:0/1300451772 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c070cf0 con 0x7f4a6c06e550 2024-03-30T07:44:49.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.939+0000 7f4a6ad9d640 1 --2- 172.21.15.47:0/1300451772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c071820 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009d90 tx=0x7f4a6402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a906d4af5ae4bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 1 -- 172.21.15.47:0/1300451772 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6402cbc0 con 0x7f4a6c06e550 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: dump: 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:49.948 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: _finish_auth 0 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: _check_auth_tickets 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 1 -- 172.21.15.47:0/1300451772 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6402cd20 con 0x7f4a6c06e550 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 5 monclient: authenticate success, global_id 17051 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 1 -- 172.21.15.47:0/1300451772 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a64034660 con 0x7f4a6c06e550 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a6959a640 10 monclient: dump: 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.949 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:49.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 4 set_mon_vals no callback set 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 10 monclient: shutdown 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 1 -- 172.21.15.47:0/1300451772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 msgr2=0x7f4a6c071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.946+0000 7f4a70b18640 1 --2- 172.21.15.47:0/1300451772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c071820 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009d90 tx=0x7f4a6402be10 comp rx=0 tx=0).stop 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 -- 172.21.15.47:0/1300451772 shutdown_connections 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 --2- 172.21.15.47:0/1300451772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c071820 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 --2- 172.21.15.47:0/1300451772 >> v2:172.21.15.47:3300/0 conn(0x7f4a6c0adad0 0x7f4a6c13ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 -- 172.21.15.47:0/1300451772 >> 172.21.15.47:0/1300451772 conn(0x7f4a6c0b30d0 msgr2=0x7f4a6c0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 -- 172.21.15.47:0/1300451772 shutdown_connections 2024-03-30T07:44:49.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.947+0000 7f4a70b18640 1 -- 172.21.15.47:0/1300451772 wait complete. 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 10 monclient: build_initial_monmap 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 10 monclient: monmap: 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 librados: starting msgr at 2024-03-30T07:44:49.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 librados: starting objecter 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 Processor -- start 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 -- start start 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 librados: setting wanted keys 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 1 librados: calling monclient init 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.948+0000 7f4a70b18640 10 monclient: init 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-c con 0x7f4a6c06e550 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6c0adad0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a6c147e50 0x7f4a6c1d6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): picked mon.noname-a con 0x7f4a6c147e50 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1477f0 con 0x7f4a6c147e50 2024-03-30T07:44:49.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0af220 con 0x7f4a6c0adad0 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1d6a40 con 0x7f4a6c06e550 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a70b18640 10 monclient(hunting): authenticate will time out at 3216.064342s 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.949+0000 7f4a69d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a6ad9d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a6c147e50 0x7f4a6c1d6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38380/0 (socket says 172.21.15.47:38380) 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 1 -- 172.21.15.47:0/3441744953 learned_addr learned my addr 172.21.15.47:0/3441744953 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): get_auth_request con 0x7f4a6c0adad0 auth_method 0 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a6ad9d640 10 monclient(hunting): get_auth_request con 0x7f4a6c147e50 auth_method 0 2024-03-30T07:44:49.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a6ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a6ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a6ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.950+0000 7f4a69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6a59c640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6a59c640 10 monclient(hunting): get_auth_request con 0x7f4a6c06e550 auth_method 0 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6a59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:49.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.951+0000 7f4a6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:49.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.952+0000 7f4a6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:49.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.952+0000 7f4a6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:49.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.952+0000 7f4a6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:49.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 10 monclient(hunting): handle_auth_done global_id 22309 payload 824 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 -- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 msgr2=0x7f4a6c147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 -- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a6c147e50 msgr2=0x7f4a6c1d6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a6c147e50 0x7f4a6c1d6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 monclient: found mon.noname-b 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.953+0000 7f4a69d9b640 1 -- 172.21.15.47:0/3441744953 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a64009a70 con 0x7f4a6c0adad0 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.955+0000 7f4a69d9b640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c4a0 tx=0x7f4a6000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.955+0000 7f4a5affd640 10 client.?.objecter ms_handle_connect 0x7f4a6c0adad0 2024-03-30T07:44:49.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.955+0000 7f4a5affd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:49.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60005ce0 con 0x7f4a6c0adad0 2024-03-30T07:44:49.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:49.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: dump: 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: _finish_auth 0 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:49.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 5 monclient: authenticate success, global_id 22309 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60005e40 con 0x7f4a6c0adad0 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 10 monclient: _renew_subs 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c13f270 con 0x7f4a6c0adad0 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a600072d0 con 0x7f4a6c0adad0 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a5affd640 10 monclient: dump: 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:49.963 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a68d99640 4 set_mon_vals no callback set 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 10 client.22309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 10 monclient: _renew_subs 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c1d7060 con 0x7f4a6c0adad0 2024-03-30T07:44:49.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.959+0000 7f4a70b18640 1 librados: init done 2024-03-30T07:44:49.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.960+0000 7f4a70b18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:49.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.960+0000 7f4a70b18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:49.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.960+0000 7f4a70b18640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:49.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.960+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c04e030 con 0x7f4a6c0adad0 2024-03-30T07:44:49.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.979+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4a600074f0 con 0x7f4a6c0adad0 2024-03-30T07:44:49.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.980+0000 7f4a5affd640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 0x7f4a4c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:49.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.980+0000 7f4a6a59c640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 0x7f4a4c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:49.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.980+0000 7f4a6a59c640 10 monclient: get_auth_request con 0x7f4a4c06c590 auth_method 0 2024-03-30T07:44:49.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.980+0000 7f4a5affd640 10 client.22309.objecter ms_handle_connect 0x7f4a4c06c590 2024-03-30T07:44:49.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.980+0000 7f4a6a59c640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 0x7f4a4c06ea50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009990 tx=0x7f4a5c009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.981+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 473..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f4a60057c40 con 0x7f4a6c0adad0 2024-03-30T07:44:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.981+0000 7f4a5affd640 10 client.22309.objecter ms_dispatch 0x7f4a6c000b90 osd_map(994..994 src has 473..994) v4 2024-03-30T07:44:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.981+0000 7f4a5affd640 3 client.22309.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.981+0000 7f4a5affd640 3 client.22309.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:49.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:49.981+0000 7f4a5affd640 20 client.22309.objecter dump_active .. 0 homeless 2024-03-30T07:44:50.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.237+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4a60053f80 con 0x7f4a6c0adad0 2024-03-30T07:44:50.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.237+0000 7f4a5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:50.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.237+0000 7f4a5affd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:50.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.424+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4a6c13f0f0 con 0x7f4a4c06c590 2024-03-30T07:44:50.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.434+0000 7f4a5affd640 1 -- 172.21.15.47:0/3441744953 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152209 (secure 0 0 0) 0x7f4a5c00e850 con 0x7f4a4c06c590 2024-03-30T07:44:50.437 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:44:50.438 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:44:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 10 librados: watch_flush enter 2024-03-30T07:44:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 10 librados: watch_flush exit 2024-03-30T07:44:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 20 client.22309.objecter shutdown clearing up homeless session... 2024-03-30T07:44:50.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 10 client.22309.objecter successfully canceled tick 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 msgr2=0x7f4a4c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 0x7f4a4c06ea50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009990 tx=0x7f4a5c009110 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 10 monclient: shutdown 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 msgr2=0x7f4a6c0ae600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c4a0 tx=0x7f4a6000c8a0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 shutdown_connections 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4a4c06c590 0x7f4a4c06ea50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a6c147e50 0x7f4a6c1d6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4a6c0adad0 0x7f4a6c0ae600 unknown :-1 s=CLOSED pgs=5502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 --2- 172.21.15.47:0/3441744953 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a6c06e550 0x7f4a6c147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 >> 172.21.15.47:0/3441744953 conn(0x7f4a6c0b30d0 msgr2=0x7f4a6c13cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 shutdown_connections 2024-03-30T07:44:50.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 -- 172.21.15.47:0/3441744953 wait complete. 2024-03-30T07:44:50.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.436+0000 7f4a70b18640 1 librados: shutdown 2024-03-30T07:44:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1848,"stamp":"2024-03-30T07:44:50.371781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529660,"kb_used_data":59172,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445060,"statfs":{"total":773094113280,"available":772551741440,"internally_reserved":0,"allocated":60592128,"data_stored":34119774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":625,"apply_latency_ms":625,"commit_latency_ns":625000000,"apply_latency_ns":625000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.254424+0000","last_change":"2024-03-30T07:18:12.759583+0000","last_active":"2024-03-30T07:44:47.254424+0000","last_peered":"2024-03-30T07:44:47.254424+0000","last_clean":"2024-03-30T07:44:47.254424+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:44:47.254424+0000","last_undegraded":"2024-03-30T07:44:47.254424+0000","last_fullsized":"2024-03-30T07:44:47.254424+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:38:53.027870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.280896+0000","last_change":"2024-03-30T07:24:14.842838+0000","last_active":"2024-03-30T07:44:47.280896+0000","last_peered":"2024-03-30T07:44:47.280896+0000","last_clean":"2024-03-30T07:44:47.280896+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:44:47.280896+0000","last_undegraded":"2024-03-30T07:44:47.280896+0000","last_fullsized":"2024-03-30T07:44:47.280896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:31:53.743297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.989999+0000","last_change":"2024-03-30T07:18:08.005915+0000","last_active":"2024-03-30T07:44:46.989999+0000","last_peered":"2024-03-30T07:44:46.989999+0000","last_clean":"2024-03-30T07:44:46.989999+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:44:46.989999+0000","last_undegraded":"2024-03-30T07:44:46.989999+0000","last_fullsized":"2024-03-30T07:44:46.989999+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:57:08.402927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":932,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.423915+0000","last_change":"2024-03-30T07:24:15.826961+0000","last_active":"2024-03-30T07:44:48.423915+0000","last_peered":"2024-03-30T07:44:48.423915+0000","last_clean":"2024-03-30T07:44:48.423915+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:44:48.423915+0000","last_undegraded":"2024-03-30T07:44:48.423915+0000","last_fullsized":"2024-03-30T07:44:48.423915+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:17:57.852075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":909,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.615585+0000","last_change":"2024-03-30T07:18:07.657409+0000","last_active":"2024-03-30T07:44:44.615585+0000","last_peered":"2024-03-30T07:44:44.615585+0000","last_clean":"2024-03-30T07:44:44.615585+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:44:44.615585+0000","last_undegraded":"2024-03-30T07:44:44.615585+0000","last_fullsized":"2024-03-30T07:44:44.615585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:39:31.459285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":920,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.617324+0000","last_change":"2024-03-30T07:44:18.246797+0000","last_active":"2024-03-30T07:44:44.617324+0000","last_peered":"2024-03-30T07:44:44.617324+0000","last_clean":"2024-03-30T07:44:44.617324+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:44:44.617324+0000","last_undegraded":"2024-03-30T07:44:44.617324+0000","last_fullsized":"2024-03-30T07:44:44.617324+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:11.652670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.113061+0000","last_change":"2024-03-30T07:44:23.336397+0000","last_active":"2024-03-30T07:44:48.113061+0000","last_peered":"2024-03-30T07:44:48.113061+0000","last_clean":"2024-03-30T07:44:48.113061+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:44:48.113061+0000","last_undegraded":"2024-03-30T07:44:48.113061+0000","last_fullsized":"2024-03-30T07:44:48.113061+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:03:38.840636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.915968+0000","last_change":"2024-03-30T07:17:58.934819+0000","last_active":"2024-03-30T07:44:46.915968+0000","last_peered":"2024-03-30T07:44:46.915968+0000","last_clean":"2024-03-30T07:44:46.915968+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:44:46.915968+0000","last_undegraded":"2024-03-30T07:44:46.915968+0000","last_fullsized":"2024-03-30T07:44:46.915968+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:40:48.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":921,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.661867+0000","last_change":"2024-03-30T07:24:15.696366+0000","last_active":"2024-03-30T07:44:44.661867+0000","last_peered":"2024-03-30T07:44:44.661867+0000","last_clean":"2024-03-30T07:44:44.661867+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:44:44.661867+0000","last_undegraded":"2024-03-30T07:44:44.661867+0000","last_fullsized":"2024-03-30T07:44:44.661867+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:36:53.871311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.646298+0000","last_change":"2024-03-30T07:18:07.825982+0000","last_active":"2024-03-30T07:44:44.646298+0000","last_peered":"2024-03-30T07:44:44.646298+0000","last_clean":"2024-03-30T07:44:44.646298+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:44.646298+0000","last_undegraded":"2024-03-30T07:44:44.646298+0000","last_fullsized":"2024-03-30T07:44:44.646298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:07:32.853655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":924,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991550+0000","last_change":"2024-03-30T07:24:13.737991+0000","last_active":"2024-03-30T07:44:46.991550+0000","last_peered":"2024-03-30T07:44:46.991550+0000","last_clean":"2024-03-30T07:44:46.991550+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:44:46.991550+0000","last_undegraded":"2024-03-30T07:44:46.991550+0000","last_fullsized":"2024-03-30T07:44:46.991550+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:44:26.609748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.654229+0000","last_change":"2024-03-30T07:18:16.996109+0000","last_active":"2024-03-30T07:44:44.654229+0000","last_peered":"2024-03-30T07:44:44.654229+0000","last_clean":"2024-03-30T07:44:44.654229+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:44:44.654229+0000","last_undegraded":"2024-03-30T07:44:44.654229+0000","last_fullsized":"2024-03-30T07:44:44.654229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:46:51.546133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":917,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.077766+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:44:47.077766+0000","last_peered":"2024-03-30T07:44:47.077766+0000","last_clean":"2024-03-30T07:44:47.077766+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:44:47.077766+0000","last_undegraded":"2024-03-30T07:44:47.077766+0000","last_fullsized":"2024-03-30T07:44:47.077766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:41:46.497967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.970317+0000","last_change":"2024-03-30T07:18:21.573712+0000","last_active":"2024-03-30T07:44:46.970317+0000","last_peered":"2024-03-30T07:44:46.970317+0000","last_clean":"2024-03-30T07:44:46.970317+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:44:46.970317+0000","last_undegraded":"2024-03-30T07:44:46.970317+0000","last_fullsized":"2024-03-30T07:44:46.970317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:18:39.319613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":913,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.650876+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:44:44.650876+0000","last_peered":"2024-03-30T07:44:44.650876+0000","last_clean":"2024-03-30T07:44:44.650876+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:44.650876+0000","last_undegraded":"2024-03-30T07:44:44.650876+0000","last_fullsized":"2024-03-30T07:44:44.650876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.663467+0000","last_change":"2024-03-30T07:18:16.070084+0000","last_active":"2024-03-30T07:44:44.663467+0000","last_peered":"2024-03-30T07:44:44.663467+0000","last_clean":"2024-03-30T07:44:44.663467+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:44:44.663467+0000","last_undegraded":"2024-03-30T07:44:44.663467+0000","last_fullsized":"2024-03-30T07:44:44.663467+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:22.115137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.979147+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:44:46.979147+0000","last_peered":"2024-03-30T07:44:46.979147+0000","last_clean":"2024-03-30T07:44:46.979147+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:44:46.979147+0000","last_undegraded":"2024-03-30T07:44:46.979147+0000","last_fullsized":"2024-03-30T07:44:46.979147+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:12.481693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.652637+0000","last_change":"2024-03-30T07:18:17.007226+0000","last_active":"2024-03-30T07:44:44.652637+0000","last_peered":"2024-03-30T07:44:44.652637+0000","last_clean":"2024-03-30T07:44:44.652637+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:44:44.652637+0000","last_undegraded":"2024-03-30T07:44:44.652637+0000","last_fullsized":"2024-03-30T07:44:44.652637+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:02:56.775071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.340367+0000","last_change":"2024-03-30T07:44:18.307851+0000","last_active":"2024-03-30T07:44:48.340367+0000","last_peered":"2024-03-30T07:44:48.340367+0000","last_clean":"2024-03-30T07:44:48.340367+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:44:48.340367+0000","last_undegraded":"2024-03-30T07:44:48.340367+0000","last_fullsized":"2024-03-30T07:44:48.340367+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:29:33.011564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991687+0000","last_change":"2024-03-30T07:17:58.343146+0000","last_active":"2024-03-30T07:44:46.991687+0000","last_peered":"2024-03-30T07:44:46.991687+0000","last_clean":"2024-03-30T07:44:46.991687+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:44:46.991687+0000","last_undegraded":"2024-03-30T07:44:46.991687+0000","last_fullsized":"2024-03-30T07:44:46.991687+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:24:03.491281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":932,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871626+0000","last_change":"2024-03-30T07:44:23.549306+0000","last_active":"2024-03-30T07:44:46.871626+0000","last_peered":"2024-03-30T07:44:46.871626+0000","last_clean":"2024-03-30T07:44:46.871626+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:44:46.871626+0000","last_undegraded":"2024-03-30T07:44:46.871626+0000","last_fullsized":"2024-03-30T07:44:46.871626+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:34:55.114766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":927,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.059001+0000","last_change":"2024-03-30T07:44:18.161552+0000","last_active":"2024-03-30T07:44:48.059001+0000","last_peered":"2024-03-30T07:44:48.059001+0000","last_clean":"2024-03-30T07:44:48.059001+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:44:48.059001+0000","last_undegraded":"2024-03-30T07:44:48.059001+0000","last_fullsized":"2024-03-30T07:44:48.059001+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:18.359722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.884996+0000","last_change":"2024-03-30T07:24:14.186380+0000","last_active":"2024-03-30T07:44:46.884996+0000","last_peered":"2024-03-30T07:44:46.884996+0000","last_clean":"2024-03-30T07:44:46.884996+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:44:46.884996+0000","last_undegraded":"2024-03-30T07:44:46.884996+0000","last_fullsized":"2024-03-30T07:44:46.884996+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:58:59.484353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.914367+0000","last_change":"2024-03-30T07:18:07.807081+0000","last_active":"2024-03-30T07:44:46.914367+0000","last_peered":"2024-03-30T07:44:46.914367+0000","last_clean":"2024-03-30T07:44:46.914367+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:44:46.914367+0000","last_undegraded":"2024-03-30T07:44:46.914367+0000","last_fullsized":"2024-03-30T07:44:46.914367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:35:37.547969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":931,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.863858+0000","last_change":"2024-03-30T07:24:13.390417+0000","last_active":"2024-03-30T07:44:46.863858+0000","last_peered":"2024-03-30T07:44:46.863858+0000","last_clean":"2024-03-30T07:44:46.863858+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:46.863858+0000","last_undegraded":"2024-03-30T07:44:46.863858+0000","last_fullsized":"2024-03-30T07:44:46.863858+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:10:59.579650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.213477+0000","last_change":"2024-03-30T07:18:17.831393+0000","last_active":"2024-03-30T07:44:47.213477+0000","last_peered":"2024-03-30T07:44:47.213477+0000","last_clean":"2024-03-30T07:44:47.213477+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:47.213477+0000","last_undegraded":"2024-03-30T07:44:47.213477+0000","last_fullsized":"2024-03-30T07:44:47.213477+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:12.485695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.954237+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:44:46.954237+0000","last_peered":"2024-03-30T07:44:46.954237+0000","last_clean":"2024-03-30T07:44:46.954237+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:46.954237+0000","last_undegraded":"2024-03-30T07:44:46.954237+0000","last_fullsized":"2024-03-30T07:44:46.954237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:47:35.848282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1075,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.140914+0000","last_change":"2024-03-30T07:24:13.767532+0000","last_active":"2024-03-30T07:44:47.140914+0000","last_peered":"2024-03-30T07:44:47.140914+0000","last_clean":"2024-03-30T07:44:47.140914+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:47.140914+0000","last_undegraded":"2024-03-30T07:44:47.140914+0000","last_fullsized":"2024-03-30T07:44:47.140914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_clean_scrub_stamp":"2024-03-30T07:16:32.203026+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:56:57.242970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":949,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.189364+0000","last_change":"2024-03-30T07:18:08.767217+0000","last_active":"2024-03-30T07:44:47.189364+0000","last_peered":"2024-03-30T07:44:47.189364+0000","last_clean":"2024-03-30T07:44:47.189364+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:44:47.189364+0000","last_undegraded":"2024-03-30T07:44:47.189364+0000","last_fullsized":"2024-03-30T07:44:47.189364+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:53:49.994420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":955,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.223130+0000","last_change":"2024-03-30T07:24:13.784512+0000","last_active":"2024-03-30T07:44:47.223130+0000","last_peered":"2024-03-30T07:44:47.223130+0000","last_clean":"2024-03-30T07:44:47.223130+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:44:47.223130+0000","last_undegraded":"2024-03-30T07:44:47.223130+0000","last_fullsized":"2024-03-30T07:44:47.223130+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:13.663028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":926,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.114701+0000","last_change":"2024-03-30T07:44:18.218063+0000","last_active":"2024-03-30T07:44:48.114701+0000","last_peered":"2024-03-30T07:44:48.114701+0000","last_clean":"2024-03-30T07:44:48.114701+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:44:48.114701+0000","last_undegraded":"2024-03-30T07:44:48.114701+0000","last_fullsized":"2024-03-30T07:44:48.114701+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:46.659263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":910,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.930337+0000","last_change":"2024-03-30T07:18:04.168467+0000","last_active":"2024-03-30T07:44:46.930337+0000","last_peered":"2024-03-30T07:44:46.930337+0000","last_clean":"2024-03-30T07:44:46.930337+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:44:46.930337+0000","last_undegraded":"2024-03-30T07:44:46.930337+0000","last_fullsized":"2024-03-30T07:44:46.930337+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:04:44.698010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":934,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871881+0000","last_change":"2024-03-30T07:24:14.248616+0000","last_active":"2024-03-30T07:44:46.871881+0000","last_peered":"2024-03-30T07:44:46.871881+0000","last_clean":"2024-03-30T07:44:46.871881+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:44:46.871881+0000","last_undegraded":"2024-03-30T07:44:46.871881+0000","last_fullsized":"2024-03-30T07:44:46.871881+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:23:34.001344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.319814+0000","last_change":"2024-03-30T07:18:53.674429+0000","last_active":"2024-03-30T07:44:47.319814+0000","last_peered":"2024-03-30T07:44:47.319814+0000","last_clean":"2024-03-30T07:44:47.319814+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:44:47.319814+0000","last_undegraded":"2024-03-30T07:44:47.319814+0000","last_fullsized":"2024-03-30T07:44:47.319814+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:04.375300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":951,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.343877+0000","last_change":"2024-03-30T07:18:37.331986+0000","last_active":"2024-03-30T07:44:47.343877+0000","last_peered":"2024-03-30T07:44:47.343877+0000","last_clean":"2024-03-30T07:44:47.343877+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:44:47.343877+0000","last_undegraded":"2024-03-30T07:44:47.343877+0000","last_fullsized":"2024-03-30T07:44:47.343877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:54:27.069173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":905,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.647899+0000","last_change":"2024-03-30T07:18:16.744594+0000","last_active":"2024-03-30T07:44:44.647899+0000","last_peered":"2024-03-30T07:44:44.647899+0000","last_clean":"2024-03-30T07:44:44.647899+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:44:44.647899+0000","last_undegraded":"2024-03-30T07:44:44.647899+0000","last_fullsized":"2024-03-30T07:44:44.647899+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:35:24.403172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.670322+0000","last_change":"2024-03-30T07:18:16.066842+0000","last_active":"2024-03-30T07:44:44.670322+0000","last_peered":"2024-03-30T07:44:44.670322+0000","last_clean":"2024-03-30T07:44:44.670322+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:44:44.670322+0000","last_undegraded":"2024-03-30T07:44:44.670322+0000","last_fullsized":"2024-03-30T07:44:44.670322+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:16:21.059197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":912,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.892761+0000","last_change":"2024-03-30T07:18:03.197722+0000","last_active":"2024-03-30T07:44:46.892761+0000","last_peered":"2024-03-30T07:44:46.892761+0000","last_clean":"2024-03-30T07:44:46.892761+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:44:46.892761+0000","last_undegraded":"2024-03-30T07:44:46.892761+0000","last_fullsized":"2024-03-30T07:44:46.892761+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:32:02.787636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":944,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.186457+0000","last_change":"2024-03-30T07:44:18.305103+0000","last_active":"2024-03-30T07:44:47.186457+0000","last_peered":"2024-03-30T07:44:47.186457+0000","last_clean":"2024-03-30T07:44:47.186457+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:44:47.186457+0000","last_undegraded":"2024-03-30T07:44:47.186457+0000","last_fullsized":"2024-03-30T07:44:47.186457+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:16:48.555361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"43'7","reported_seq":941,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.091873+0000","last_change":"2024-03-30T07:44:18.289552+0000","last_active":"2024-03-30T07:44:48.091873+0000","last_peered":"2024-03-30T07:44:48.091873+0000","last_clean":"2024-03-30T07:44:48.091873+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:44:48.091873+0000","last_undegraded":"2024-03-30T07:44:48.091873+0000","last_fullsized":"2024-03-30T07:44:48.091873+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:06:03.894404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.376658+0000","last_change":"2024-03-30T07:24:15.802335+0000","last_active":"2024-03-30T07:44:48.376658+0000","last_peered":"2024-03-30T07:44:48.376658+0000","last_clean":"2024-03-30T07:44:48.376658+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:44:48.376658+0000","last_undegraded":"2024-03-30T07:44:48.376658+0000","last_fullsized":"2024-03-30T07:44:48.376658+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:04:34.288781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64668,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307172,"statfs":{"total":96636764160,"available":96570544128,"internally_reserved":0,"allocated":7487488,"data_stored":4185223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:25 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":18.387,"15min":18.98},"min":{"1min":5.0540000000000003,"5min":4.7649999999999997,"15min":4.7649999999999997},"max":{"1min":12.066000000000001,"5min":249.63200000000001,"15min":292.04199999999997},"last":6.2889999999999997},{"interface":"front","average":{"1min":12.994,"5min":21.666,"15min":20.100000000000001},"min":{"1min":6.0720000000000001,"5min":5.0730000000000004,"15min":4.79},"max":{"1min":39.451999999999998,"5min":248.767,"15min":299.35000000000002},"last":15.112}]},{"osd":1,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":10.112,"5min":17.547999999999998,"15min":18.082000000000001},"min":{"1min":6.3440000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":20.783999999999999,"5min":169.17099999999999,"15min":169.17099999999999},"last":7.4710000000000001},{"interface":"front","average":{"1min":12.683999999999999,"5min":24.754000000000001,"15min":25.129000000000001},"min":{"1min":8.4939999999999998,"5min":7.8929999999999998,"15min":6.7789999999999999},"max":{"1min":15.541,"5min":183.77799999999999,"15min":286.52199999999999},"last":12.396000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1180.998,"5min":530.18200000000002,"15min":228.39699999999999},"min":{"1min":9.0640000000000001,"5min":9.0640000000000001,"15min":8.0619999999999994},"max":{"1min":14020.602000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":20.991},{"interface":"front","average":{"1min":1157.28,"5min":537.84500000000003,"15min":233.36799999999999},"min":{"1min":12.426,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13714.877,"5min":13714.877,"15min":13714.877},"last":19.504000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":11.715999999999999,"5min":335.99599999999998,"15min":275.75700000000001},"min":{"1min":7.665,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":16.957999999999998,"5min":12556.546,"15min":13820.906000000001},"last":11.281000000000001},{"interface":"front","average":{"1min":14.755000000000001,"5min":341.73700000000002,"15min":281.92700000000002},"min":{"1min":9.9740000000000002,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":18.872,"5min":12557.409,"15min":13821.846},"last":15.805999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":14.395,"5min":28.911999999999999,"15min":50.296999999999997},"min":{"1min":12.161,"5min":10.385,"15min":9.8539999999999992},"max":{"1min":17.561,"5min":254.04300000000001,"15min":4768.4960000000001},"last":17.469000000000001},{"interface":"front","average":{"1min":16.638999999999999,"5min":35.829000000000001,"15min":50.470999999999997},"min":{"1min":13.890000000000001,"5min":13.061999999999999,"15min":7.8760000000000003},"max":{"1min":21.492000000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":20.231999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:42:20 2024","interfaces":[{"interface":"back","average":{"1min":57.975000000000001,"5min":40.822000000000003,"15min":28.888000000000002},"min":{"1min":14.178000000000001,"5min":12.103,"15min":6.9530000000000003},"max":{"1min":328.01600000000002,"5min":328.01600000000002,"15min":350.94400000000002},"last":14.252000000000001},{"interface":"front","average":{"1min":32.357999999999997,"5min":32.185000000000002,"15min":29.07},"min":{"1min":9.5039999999999996,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":98.340999999999994,"5min":294.49700000000001,"15min":323.77999999999997},"last":18.573}]},{"osd":7,"last update":"Sat Mar 30 07:44:07 2024","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":21.923999999999999,"15min":337.53199999999998},"min":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":19.643999999999998,"5min":334.75400000000002,"15min":19293.749},"last":12.718999999999999},{"interface":"front","average":{"1min":14.932,"5min":27.75,"15min":339.43299999999999},"min":{"1min":10.574999999999999,"5min":9.1430000000000007,"15min":8.2799999999999994},"max":{"1min":20.895,"5min":318.11200000000002,"15min":19292.978999999999},"last":11.971}]}]},{"osd":0,"up_from":22,"seq":94489280857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60540,"kb_used_data":7276,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311300,"statfs":{"total":96636764160,"available":96574771200,"internally_reserved":0,"allocated":7450624,"data_stored":4139468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":132.09700000000001},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":159.65100000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":32.520000000000003},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":36.177}]},{"osd":3,"last update":"Sat Mar 30 07:43:47 2024","interfaces":[{"interface":"back","average":{"1min":16.346,"5min":178.83799999999999,"15min":96.840000000000003},"min":{"1min":11.311,"5min":11.311,"15min":6.6429999999999998},"max":{"1min":36.655000000000001,"5min":9964.1550000000007,"15min":9964.1550000000007},"last":26.074999999999999},{"interface":"front","average":{"1min":12.576000000000001,"5min":166.869,"15min":92.548000000000002},"min":{"1min":8.0670000000000002,"5min":5.2569999999999997,"15min":5.2569999999999997},"max":{"1min":33.716000000000001,"5min":9945.7579999999998,"15min":9945.7579999999998},"last":33.866999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":168.22499999999999},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":134.59100000000001}]},{"osd":5,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":12.333,"5min":20.417999999999999,"15min":59.168999999999997},"min":{"1min":8.4209999999999994,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":17.195,"5min":298.24599999999998,"15min":7478.7280000000001},"last":35.418999999999997},{"interface":"front","average":{"1min":16.745000000000001,"5min":33.078000000000003,"15min":60.576999999999998},"min":{"1min":11.897,"5min":11.897,"15min":7.125},"max":{"1min":21.306000000000001,"5min":383.13999999999999,"15min":7477.2579999999998},"last":29.375}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":44.006999999999998,"5min":37.201999999999998,"15min":29.158999999999999},"min":{"1min":13.180999999999999,"5min":12.316000000000001,"15min":8.7569999999999997},"max":{"1min":386.452,"5min":386.452,"15min":386.452},"last":38.423000000000002},{"interface":"front","average":{"1min":31.984999999999999,"5min":28.108000000000001,"15min":25.620999999999999},"min":{"1min":8.2970000000000006,"5min":7.835,"15min":7.835},"max":{"1min":300.58199999999999,"5min":300.58199999999999,"15min":300.58199999999999},"last":37.646999999999998}]},{"osd":7,"last update":"Sat Mar 30 07:44:06 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":22.975999999999999,"15min":327.85500000000002},"min":{"1min":7.7320000000000002,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":18.007000000000001,"5min":295.38600000000002,"15min":21868.147000000001},"last":30.052},{"interface":"front","average":{"1min":13.711,"5min":24.344000000000001,"15min":331.459},"min":{"1min":10.34,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":18.896999999999998,"5min":302.11500000000001,"15min":21842.699000000001},"last":48.261000000000003}]}]},{"osd":4,"up_from":20,"seq":85899346251,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78628,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293212,"statfs":{"total":96636764160,"available":96556249088,"internally_reserved":0,"allocated":7487488,"data_stored":4198149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":6.7249999999999996,"5min":24.440999999999999,"15min":22.800999999999998},"min":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":13.228,"5min":350.36599999999999,"15min":759.322},"last":172.70699999999999},{"interface":"front","average":{"1min":8.5169999999999995,"5min":27.239000000000001,"15min":28.228000000000002},"min":{"1min":5.9379999999999997,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":11.881,"5min":410.93099999999998,"15min":761.40899999999999},"last":53.140999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":10.500999999999999,"5min":21.305,"15min":21.925000000000001},"min":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":26.32,"5min":313.54399999999998,"15min":313.54399999999998},"last":65.491},{"interface":"front","average":{"1min":13.638999999999999,"5min":29.907,"15min":29.178000000000001},"min":{"1min":8.8420000000000005,"5min":8.8420000000000005,"15min":6.3719999999999999},"max":{"1min":27.036999999999999,"5min":466.54599999999999,"15min":466.54599999999999},"last":66.787000000000006}]},{"osd":2,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":9.2159999999999993,"5min":21.456,"15min":26.949999999999999},"min":{"1min":6.3659999999999997,"5min":6.1139999999999999,"15min":5.7800000000000002},"max":{"1min":11.869,"5min":371.74200000000002,"15min":768.79100000000005},"last":33.648000000000003},{"interface":"front","average":{"1min":13.162000000000001,"5min":25.972999999999999,"15min":23.812999999999999},"min":{"1min":9.3089999999999993,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":17.018000000000001,"5min":372.322,"15min":372.322},"last":37.347000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":12.989000000000001,"5min":413.31400000000002,"15min":188.03399999999999},"min":{"1min":8.3800000000000008,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":17.893999999999998,"5min":11317.004000000001,"15min":11317.004000000001},"last":46.061999999999998},{"interface":"front","average":{"1min":15.349,"5min":405.84899999999999,"15min":187.77000000000001},"min":{"1min":8.4019999999999992,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":46.728999999999999,"5min":11222.611999999999,"15min":11222.611999999999},"last":173.566}]},{"osd":5,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":13.702,"5min":27.013000000000002,"15min":43.438000000000002},"min":{"1min":10.589,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":16.962,"5min":421.40600000000001,"15min":4807.1080000000002},"last":88.423000000000002},{"interface":"front","average":{"1min":14.109999999999999,"5min":28.047000000000001,"15min":47.658000000000001},"min":{"1min":8.6690000000000005,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":17.809999999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":73.852000000000004}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":35.414000000000001,"5min":31.486000000000001,"15min":30.466999999999999},"min":{"1min":9.8079999999999998,"5min":9.4640000000000004,"15min":9.4640000000000004},"max":{"1min":223.85499999999999,"5min":529.13699999999994,"15min":529.13699999999994},"last":72.775999999999996},{"interface":"front","average":{"1min":38.140999999999998,"5min":34.231999999999999,"15min":36.503999999999998},"min":{"1min":13.925000000000001,"5min":11.499000000000001,"15min":7.1870000000000003},"max":{"1min":223.16999999999999,"5min":529.73000000000002,"15min":967.58399999999995},"last":118.96299999999999}]},{"osd":7,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":14.84,"5min":32.366999999999997,"15min":341.73399999999998},"min":{"1min":10.010999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":19.140999999999998,"5min":530.32500000000005,"15min":20845.164000000001},"last":30.667000000000002},{"interface":"front","average":{"1min":16.094999999999999,"5min":41.709000000000003,"15min":346.267},"min":{"1min":9.3520000000000003,"5min":8.1620000000000008,"15min":7.5060000000000002},"max":{"1min":19.088000000000001,"5min":664.02300000000002,"15min":20888.814999999999},"last":29.001000000000001}]}]},{"osd":7,"up_from":19,"seq":81604378951,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60604,"kb_used_data":7336,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311236,"statfs":{"total":96636764160,"available":96574705664,"internally_reserved":0,"allocated":7512064,"data_stored":4200908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":21.870999999999999,"15min":22.189},"min":{"1min":5.0609999999999999,"5min":4.8719999999999999,"15min":4.2789999999999999},"max":{"1min":51.43,"5min":358.94499999999999,"15min":368.16899999999998},"last":12.932},{"interface":"front","average":{"1min":11.212999999999999,"5min":25.902000000000001,"15min":24.701000000000001},"min":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":50.631,"5min":349.97199999999998,"15min":349.97199999999998},"last":6.8639999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":9.4149999999999991,"5min":14.821,"15min":24.390999999999998},"min":{"1min":4.577,"5min":4.577,"15min":4.577},"max":{"1min":16.036999999999999,"5min":94.453999999999994,"15min":472.33300000000003},"last":7.5700000000000003},{"interface":"front","average":{"1min":11.449,"5min":21.518000000000001,"15min":26.239000000000001},"min":{"1min":7.4370000000000003,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":14.566000000000001,"5min":184.33500000000001,"15min":292.92099999999999},"last":12.113}]},{"osd":2,"last update":"Sat Mar 30 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":15.978999999999999,"5min":27.795000000000002,"15min":31.538},"min":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":6.6449999999999996},"max":{"1min":77.424000000000007,"5min":341.96100000000001,"15min":400.43099999999998},"last":13.747},{"interface":"front","average":{"1min":17.446000000000002,"5min":33.427,"15min":34.75},"min":{"1min":10.714,"5min":10.554,"15min":7.5049999999999999},"max":{"1min":53.476999999999997,"5min":389.37200000000001,"15min":439.31999999999999},"last":15.32}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":896.39800000000002,"5min":450.06999999999999,"15min":204.417},"min":{"1min":11.375999999999999,"5min":11.375999999999999,"15min":7.5940000000000003},"max":{"1min":13240.941000000001,"5min":13240.941000000001,"15min":13240.941000000001},"last":14.526},{"interface":"front","average":{"1min":915.66300000000001,"5min":450.815,"15min":204.88300000000001},"min":{"1min":7.484,"5min":7.423,"15min":7.423},"max":{"1min":13596.368,"5min":13596.368,"15min":13596.368},"last":9.5570000000000004}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":11.497,"5min":349.65699999999998,"15min":225.608},"min":{"1min":7.0609999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":19.181999999999999,"5min":11792.466,"15min":13877.913},"last":16.128},{"interface":"front","average":{"1min":13.494999999999999,"5min":360.642,"15min":230.58600000000001},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":19.454000000000001,"5min":11966.236999999999,"15min":14034.805},"last":8.9589999999999996}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":24.239000000000001,"15min":108.086},"min":{"1min":11.356999999999999,"5min":7.6619999999999999,"15min":7.4050000000000002},"max":{"1min":21.242000000000001,"5min":345.90499999999997,"15min":7694.1369999999997},"last":17.125},{"interface":"front","average":{"1min":12.236000000000001,"5min":22.170000000000002,"15min":112.55800000000001},"min":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":18.193000000000001,"5min":203.31299999999999,"15min":7725.4070000000002},"last":11.071}]},{"osd":6,"last update":"Sat Mar 30 07:42:11 2024","interfaces":[{"interface":"back","average":{"1min":27.544,"5min":35.518999999999998,"15min":31.760999999999999},"min":{"1min":9.8190000000000008,"5min":8.7620000000000005,"15min":8.7620000000000005},"max":{"1min":171.96100000000001,"5min":316.012,"15min":316.012},"last":18.978000000000002},{"interface":"front","average":{"1min":35.613999999999997,"5min":36.930999999999997,"15min":29.056999999999999},"min":{"1min":18.420999999999999,"5min":6.8700000000000001,"15min":6.3090000000000002},"max":{"1min":194.72399999999999,"5min":743.27700000000004,"15min":743.27700000000004},"last":18.029}]}]},{"osd":1,"up_from":16,"seq":68719477077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61016,"kb_used_data":7672,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310824,"statfs":{"total":96636764160,"available":96574283776,"internally_reserved":0,"allocated":7856128,"data_stored":4537313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":10.23,"5min":13.962,"15min":20.478000000000002},"min":{"1min":5.5880000000000001,"5min":5.2140000000000004,"15min":5.077},"max":{"1min":16.25,"5min":132.02699999999999,"15min":421.78699999999998},"last":6.0880000000000001},{"interface":"front","average":{"1min":9.8460000000000001,"5min":13.878,"15min":20.594999999999999},"min":{"1min":5.3019999999999996,"5min":5.2759999999999998,"15min":5.1139999999999999},"max":{"1min":13.212999999999999,"5min":116.801,"15min":409.91000000000003},"last":8.5020000000000007}]},{"osd":2,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":14.778,"15min":21.677},"min":{"1min":5.7489999999999997,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":18.859999999999999,"5min":180.42599999999999,"15min":433.34899999999999},"last":7.04},{"interface":"front","average":{"1min":11.426,"5min":18.071999999999999,"15min":23.861000000000001},"min":{"1min":8.8480000000000008,"5min":7.0700000000000003,"15min":5.6619999999999999},"max":{"1min":15.522,"5min":177.982,"15min":624.27300000000002},"last":15.009}]},{"osd":3,"last update":"Sat Mar 30 07:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1671.011,"5min":557.96100000000001,"15min":225.822},"min":{"1min":9.8309999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":12913.004000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":16.109999999999999},{"interface":"front","average":{"1min":1712.2249999999999,"5min":566.452,"15min":227.315},"min":{"1min":9.1899999999999995,"5min":7.1689999999999996,"15min":6.6130000000000004},"max":{"1min":13136.255999999999,"5min":13136.255999999999,"15min":13136.255999999999},"last":9.0869999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":18.109000000000002},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":9.6419999999999995}]},{"osd":5,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":17.719000000000001,"5min":20.428999999999998,"15min":47.131},"min":{"1min":12.308999999999999,"5min":9.8079999999999998,"15min":8.3439999999999994},"max":{"1min":44.511000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":17.315999999999999},{"interface":"front","average":{"1min":16.501999999999999,"5min":19.215,"15min":44.482999999999997},"min":{"1min":11.026,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":46.036000000000001,"5min":140.625,"15min":7345.2740000000003},"last":10.175000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:41:46 2024","interfaces":[{"interface":"back","average":{"1min":17.646999999999998,"5min":19.326000000000001,"15min":26.009},"min":{"1min":11.462,"5min":9.2889999999999997,"15min":7.8540000000000001},"max":{"1min":24.370000000000001,"5min":137.40199999999999,"15min":318.84100000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":12.010999999999999,"5min":13.401999999999999,"15min":24.765000000000001},"min":{"1min":8.1199999999999992,"5min":7.7110000000000003,"15min":7.4210000000000003},"max":{"1min":18.332000000000001,"5min":71.405000000000001,"15min":452.00700000000001},"last":7.774}]},{"osd":7,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":14.731999999999999,"5min":16.477,"15min":243.119},"min":{"1min":9.6280000000000001,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":20.175999999999998,"5min":115.461,"15min":20279.343000000001},"last":18.882000000000001},{"interface":"front","average":{"1min":12.869999999999999,"5min":22.413,"15min":251.726},"min":{"1min":6.4550000000000001,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":17.719999999999999,"5min":281.423,"15min":20311.815999999999},"last":10.976000000000001}]}]},{"osd":5,"up_from":231,"seq":992137445697,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":10.680999999999999},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":6.3929999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":21.510000000000002},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":11.289}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":22.34},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":15.675000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":7.6020000000000003},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":23.135999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":11.842000000000001},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":17.524000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:38 2024","interfaces":[{"interface":"back","average":{"1min":30.094999999999999,"5min":22.585000000000001,"15min":27.227},"min":{"1min":14.298999999999999,"5min":11.044,"15min":8.9009999999999998},"max":{"1min":102.39400000000001,"5min":102.39400000000001,"15min":236.04599999999999},"last":16.920000000000002},{"interface":"front","average":{"1min":21.274000000000001,"5min":17.177,"15min":24.478999999999999},"min":{"1min":10.237,"5min":9.1620000000000008,"15min":6.4249999999999998},"max":{"1min":85.435000000000002,"5min":85.435000000000002,"15min":248.31299999999999},"last":25.350000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:13 2024","interfaces":[{"interface":"back","average":{"1min":11.587,"5min":15.007,"15min":283.82900000000001},"min":{"1min":7.585,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":18.324999999999999,"5min":115.589,"15min":19723.973000000002},"last":20.396000000000001},{"interface":"front","average":{"1min":15.435,"5min":19.620999999999999,"15min":283.815},"min":{"1min":10.228999999999999,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":19.495000000000001,"5min":95.783000000000001,"15min":19724.860000000001},"last":26.346}]}]},{"osd":3,"up_from":14,"seq":60129542477,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82692,"kb_used_data":7280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289148,"statfs":{"total":96636764160,"available":96552087552,"internally_reserved":0,"allocated":7454720,"data_stored":4161164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":38.729999999999997},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":226.017}]},{"osd":1,"last update":"Sat Mar 30 07:44:28 2024","interfaces":[{"interface":"back","average":{"1min":34.786999999999999,"5min":22.965,"15min":21.632999999999999},"min":{"1min":8.6310000000000002,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":291.73599999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":227.48699999999999},{"interface":"front","average":{"1min":48.518999999999998,"5min":29.795999999999999,"15min":25.812000000000001},"min":{"1min":7.7089999999999996,"5min":7.5599999999999996,"15min":6.415},"max":{"1min":541.21199999999999,"5min":541.21199999999999,"15min":541.21199999999999},"last":159.37200000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":148.59399999999999},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":225.03899999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":158.285},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":248.155}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":231.13},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":278.62700000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:36 2024","interfaces":[{"interface":"back","average":{"1min":21.367999999999999,"5min":31.001999999999999,"15min":27.120000000000001},"min":{"1min":11.406000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":110.607,"5min":435.96600000000001,"15min":435.96600000000001},"last":172.33199999999999},{"interface":"front","average":{"1min":21.904,"5min":24.468,"15min":25.545999999999999},"min":{"1min":10.824,"5min":10.608000000000001,"15min":8.1180000000000003},"max":{"1min":62.828000000000003,"5min":145.239,"15min":259.60399999999998},"last":248.833}]},{"osd":7,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":39.734000000000002,"5min":26.948,"15min":285.68700000000001},"min":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":8.2690000000000001},"max":{"1min":527.00300000000004,"5min":527.00300000000004,"15min":20495.457999999999},"last":295.31599999999997},{"interface":"front","average":{"1min":30.413,"5min":28.954000000000001,"15min":287.50799999999998},"min":{"1min":11.025,"5min":11.025,"15min":11.025},"max":{"1min":308.75900000000001,"5min":308.75900000000001,"15min":20496.241000000002},"last":262.07799999999997}]}]},{"osd":6,"up_from":978,"seq":4200478015811,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7292,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7467008,"data_stored":4146127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":463.66800000000001,"5min":104.768,"15min":47.588000000000001},"min":{"1min":463.66800000000001,"5min":5.8390000000000004,"15min":4.6150000000000002},"max":{"1min":463.66800000000001,"5min":463.66800000000001,"15min":463.66800000000001},"last":11.76},{"interface":"front","average":{"1min":466.404,"5min":108.209,"15min":52.970999999999997},"min":{"1min":466.404,"5min":8.5449999999999999,"15min":5.8150000000000004},"max":{"1min":466.404,"5min":466.404,"15min":466.404},"last":7.5309999999999997}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":45.661999999999999,"5min":19.164999999999999,"15min":17.347000000000001},"min":{"1min":6.5149999999999997,"5min":5.4669999999999996,"15min":4.8579999999999997},"max":{"1min":298.72500000000002,"5min":298.72500000000002,"15min":298.72500000000002},"last":14.444000000000001},{"interface":"front","average":{"1min":53.417000000000002,"5min":25.199999999999999,"15min":22.443000000000001},"min":{"1min":8.4190000000000005,"5min":7.944,"15min":7.5519999999999996},"max":{"1min":323.78800000000001,"5min":323.78800000000001,"15min":323.78800000000001},"last":12.382}]},{"osd":2,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":314.89600000000002,"5min":76.939999999999998,"15min":39.972000000000001},"min":{"1min":314.89600000000002,"5min":6.351,"15min":6.351},"max":{"1min":314.89600000000002,"5min":314.89600000000002,"15min":314.89600000000002},"last":29.376999999999999},{"interface":"front","average":{"1min":353.24000000000001,"5min":87.257999999999996,"15min":46.240000000000002},"min":{"1min":353.24000000000001,"5min":8.266,"15min":8.266},"max":{"1min":353.24000000000001,"5min":353.24000000000001,"15min":353.24000000000001},"last":36.253}]},{"osd":3,"last update":"Sat Mar 30 07:42:29 2024","interfaces":[{"interface":"back","average":{"1min":22.446999999999999,"5min":188.88,"15min":130.78899999999999},"min":{"1min":9.8559999999999999,"5min":6.7830000000000004,"15min":6.7830000000000004},"max":{"1min":144.38300000000001,"5min":9266.1270000000004,"15min":9266.1270000000004},"last":8.4589999999999996},{"interface":"front","average":{"1min":16.998000000000001,"5min":177.846,"15min":123.959},"min":{"1min":7.3179999999999996,"5min":6.1200000000000001,"15min":4.8339999999999996},"max":{"1min":86.635000000000005,"5min":9082.3240000000005,"15min":9082.3240000000005},"last":18.457999999999998}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":331.70499999999998,"5min":557.46400000000006,"15min":326.00999999999999},"min":{"1min":331.70499999999998,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":331.70499999999998,"5min":13752.589,"15min":13752.589},"last":16.968},{"interface":"front","average":{"1min":469.358,"5min":575.79200000000003,"15min":334.21499999999997},"min":{"1min":469.358,"5min":7.5999999999999996,"15min":7.0199999999999996},"max":{"1min":469.358,"5min":13613.244000000001,"15min":13613.244000000001},"last":13.032999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":42.067999999999998,"5min":19.300999999999998,"15min":110.551},"min":{"1min":8.1010000000000009,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":483.904,"5min":483.904,"15min":9617.3179999999993},"last":16.385999999999999},{"interface":"front","average":{"1min":46.238,"5min":24.486000000000001,"15min":110.789},"min":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":8.8580000000000005},"max":{"1min":485.661,"5min":485.661,"15min":9386.0400000000009},"last":9.8219999999999992}]},{"osd":7,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":67.293000000000006,"5min":34.445,"15min":29.379999999999999},"min":{"1min":12.813000000000001,"5min":7.9729999999999999,"15min":7.9729999999999999},"max":{"1min":470.34300000000002,"5min":470.34300000000002,"15min":470.34300000000002},"last":17.635000000000002},{"interface":"front","average":{"1min":70.335999999999999,"5min":35.420000000000002,"15min":29.364000000000001},"min":{"1min":12.48,"5min":8.452,"15min":7.4189999999999996},"max":{"1min":487.30700000000002,"5min":487.30700000000002,"15min":487.30700000000002},"last":15.579000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:44:50.452 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.552+0000 7fac9a91b640 10 monclient: get_monmap_and_config 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.552+0000 7fac9a91b640 10 monclient: build_initial_monmap 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.552+0000 7fac9a91b640 10 monclient: monmap: 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:50.553040+0000 2024-03-30T07:44:50.554 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:50.553040+0000 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:50.555 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 Processor -- start 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 -- start start 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient: init 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-30T07:44:50.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac940b4280 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac9413e930 0x7fac9413eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-a con 0x7fac9413e930 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fac94141670 0x7fac94141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-c con 0x7fac94141670 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9413f2e0 con 0x7fac940b4280 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940b4660 con 0x7fac9413e930 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940aa3c0 con 0x7fac94141670 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.553+0000 7fac9a91b640 10 monclient(hunting): authenticate will time out at 3216.668823s 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac937fe640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fac9413e930 0x7fac9413eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac98eb8640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fac94141670 0x7fac94141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac93fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fac940b4280 0x7fac940a8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:33612/0 (socket says 172.21.15.47:33612) 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac93fff640 1 -- 172.21.15.47:0/433788869 learned_addr learned my addr 172.21.15.47:0/433788869 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac937fe640 10 monclient(hunting): get_auth_request con 0x7fac9413e930 auth_method 0 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac937fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac98eb8640 10 monclient(hunting): get_auth_request con 0x7fac94141670 auth_method 0 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac98eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac98eb8640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.554+0000 7fac98eb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:50.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.555+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:50.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 10 monclient(hunting): handle_auth_done global_id 15816 payload 274 2024-03-30T07:44:50.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 1 -- 172.21.15.47:0/433788869 >> v2:172.21.15.52:3300/0 conn(0x7fac94141670 msgr2=0x7fac94141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:50.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 1 --2- 172.21.15.47:0/433788869 >> v2:172.21.15.52:3300/0 conn(0x7fac94141670 0x7fac94141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 1 -- 172.21.15.47:0/433788869 >> v1:172.21.15.52:6789/0 conn(0x7fac940b4280 legacy=0x7fac940a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 10 monclient: _finish_hunting 0 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 1 monclient: found mon.noname-a 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.556+0000 7fac937fe640 1 -- 172.21.15.47:0/433788869 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940aa630 con 0x7fac9413e930 2024-03-30T07:44:50.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.558+0000 7fac937fe640 1 --2- 172.21.15.47:0/433788869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac9413e930 0x7fac9413eda0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7fac940b39e0 tx=0x7fac8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4767815a9b1e0380 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:50.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 1 -- 172.21.15.47:0/433788869 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8802e070 con 0x7fac9413e930 2024-03-30T07:44:50.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:50.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: dump: 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: _finish_auth 0 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:50.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 1 -- 172.21.15.47:0/433788869 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac880315a0 con 0x7fac9413e930 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 1 -- 172.21.15.47:0/433788869 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88031930 con 0x7fac9413e930 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac92ffd640 10 monclient: dump: 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:50.567 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac9a91b640 5 monclient: authenticate success, global_id 15816 2024-03-30T07:44:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac9a91b640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.562+0000 7fac9a91b640 4 set_mon_vals no callback set 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 10 monclient: shutdown 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 -- 172.21.15.47:0/433788869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac9413e930 msgr2=0x7fac9413eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 --2- 172.21.15.47:0/433788869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac9413e930 0x7fac9413eda0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7fac940b39e0 tx=0x7fac8802b840 comp rx=0 tx=0).stop 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 -- 172.21.15.47:0/433788869 shutdown_connections 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 --2- 172.21.15.47:0/433788869 >> v2:172.21.15.52:3300/0 conn(0x7fac94141670 0x7fac94141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 --2- 172.21.15.47:0/433788869 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac9413e930 0x7fac9413eda0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 -- 172.21.15.47:0/433788869 >> 172.21.15.47:0/433788869 conn(0x7fac94065e00 msgr2=0x7fac94066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 -- 172.21.15.47:0/433788869 shutdown_connections 2024-03-30T07:44:50.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.563+0000 7fac9a91b640 1 -- 172.21.15.47:0/433788869 wait complete. 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 10 monclient: build_initial_monmap 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 10 monclient: monmap: 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 librados: starting msgr at 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 librados: starting objecter 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 Processor -- start 2024-03-30T07:44:50.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 -- start start 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 librados: setting wanted keys 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 1 librados: calling monclient init 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.564+0000 7fac9a91b640 10 monclient: init 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac940b4280 0x7fac941d9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-c con 0x7fac940b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-b con 0x7fac9413e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac94141670 0x7fac9413caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): picked mon.noname-a con 0x7fac94141670 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9413c6d0 con 0x7fac94141670 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:50.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941321e0 con 0x7fac9413e930 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9413c3c0 con 0x7fac940b4280 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac9a91b640 10 monclient(hunting): authenticate will time out at 3216.680300s 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac93fff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac940b4280 0x7fac941d9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac98eb8640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac94141670 0x7fac9413caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38402/0 (socket says 172.21.15.47:38402) 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 1 -- 172.21.15.47:0/3321920019 learned_addr learned my addr 172.21.15.47:0/3321920019 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 10 monclient(hunting): get_auth_request con 0x7fac9413e930 auth_method 0 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.565+0000 7fac937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): get_auth_request con 0x7fac940b4280 auth_method 0 2024-03-30T07:44:50.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): get_auth_request con 0x7fac94141670 auth_method 0 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:50.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.566+0000 7fac98eb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 10 monclient(hunting): handle_auth_done global_id 22315 payload 824 2024-03-30T07:44:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 -- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac940b4280 msgr2=0x7fac941d9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac940b4280 0x7fac941d9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 -- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac94141670 msgr2=0x7fac9413caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac94141670 0x7fac9413caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 10 monclient: _finish_hunting 0 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 monclient: found mon.noname-b 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.569+0000 7fac937fe640 1 -- 172.21.15.47:0/3321920019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac88009400 con 0x7fac9413e930 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.571+0000 7fac937fe640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 secure :-1 s=READY pgs=5504 cs=0 l=1 rev1=1 crypto rx=0x7fac88009890 tx=0x7fac880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.571+0000 7fac90ff9640 10 client.?.objecter ms_handle_connect 0x7fac9413e930 2024-03-30T07:44:50.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.571+0000 7fac90ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88032840 con 0x7fac9413e930 2024-03-30T07:44:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:50.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: dump: 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: _finish_auth 0 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:50.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac880329c0 con 0x7fac9413e930 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88032d70 con 0x7fac9413e930 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac90ff9640 10 monclient: dump: 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 5 monclient: authenticate success, global_id 22315 2024-03-30T07:44:50.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 10 monclient: _renew_subs 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac941da700 con 0x7fac9413e930 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac927fc640 4 set_mon_vals no callback set 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 10 client.22315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 10 monclient: _renew_subs 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9413d210 con 0x7fac9413e930 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.574+0000 7fac9a91b640 1 librados: init done 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.575+0000 7fac9a91b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.575+0000 7fac9a91b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.575+0000 7fac9a91b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:50.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.575+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac941da8c0 con 0x7fac9413e930 2024-03-30T07:44:50.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.595+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fac8804e470 con 0x7fac9413e930 2024-03-30T07:44:50.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.595+0000 7fac90ff9640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 0x7fac8006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:50.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.595+0000 7fac93fff640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 0x7fac8006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:50.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.595+0000 7fac93fff640 10 monclient: get_auth_request con 0x7fac8006c5e0 auth_method 0 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 10 client.22315.objecter ms_handle_connect 0x7fac8006c5e0 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac93fff640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 0x7fac8006eaa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fac84000f60 tx=0x7fac84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 473..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fac8807a100 con 0x7fac9413e930 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 10 client.22315.objecter ms_dispatch 0x7fac94000b90 osd_map(994..994 src has 473..994) v4 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 3 client.22315.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:50.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 3 client.22315.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:50.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.596+0000 7fac90ff9640 20 client.22315.objecter dump_active .. 0 homeless 2024-03-30T07:44:50.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.851+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fac88076440 con 0x7fac9413e930 2024-03-30T07:44:50.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.851+0000 7fac90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:50.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:50.851+0000 7fac90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:51.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.055+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac9404e030 con 0x7fac8006c5e0 2024-03-30T07:44:51.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.060+0000 7fac90ff9640 1 -- 172.21.15.47:0/3321920019 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152209 (secure 0 0 0) 0x7fac9404e030 con 0x7fac8006c5e0 2024-03-30T07:44:51.064 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:44:51.064 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 10 librados: watch_flush enter 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 10 librados: watch_flush exit 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 20 client.22315.objecter shutdown clearing up homeless session... 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 10 client.22315.objecter successfully canceled tick 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 msgr2=0x7fac8006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 0x7fac8006eaa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fac84000f60 tx=0x7fac84009040 comp rx=0 tx=0).stop 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 10 monclient: shutdown 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:51.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 msgr2=0x7fac941da160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 secure :-1 s=READY pgs=5504 cs=0 l=1 rev1=1 crypto rx=0x7fac88009890 tx=0x7fac880046c0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 shutdown_connections 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fac8006c5e0 0x7fac8006eaa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fac94141670 0x7fac9413caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fac9413e930 0x7fac941da160 unknown :-1 s=CLOSED pgs=5504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 --2- 172.21.15.47:0/3321920019 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fac940b4280 0x7fac941d9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 >> 172.21.15.47:0/3321920019 conn(0x7fac94065e00 msgr2=0x7fac940b5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 shutdown_connections 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 -- 172.21.15.47:0/3321920019 wait complete. 2024-03-30T07:44:51.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.063+0000 7fac9a91b640 1 librados: shutdown 2024-03-30T07:44:51.075 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1848,"stamp":"2024-03-30T07:44:50.371781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529660,"kb_used_data":59172,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445060,"statfs":{"total":773094113280,"available":772551741440,"internally_reserved":0,"allocated":60592128,"data_stored":34119774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":625,"apply_latency_ms":625,"commit_latency_ns":625000000,"apply_latency_ns":625000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.254424+0000","last_change":"2024-03-30T07:18:12.759583+0000","last_active":"2024-03-30T07:44:47.254424+0000","last_peered":"2024-03-30T07:44:47.254424+0000","last_clean":"2024-03-30T07:44:47.254424+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:44:47.254424+0000","last_undegraded":"2024-03-30T07:44:47.254424+0000","last_fullsized":"2024-03-30T07:44:47.254424+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:38:53.027870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.280896+0000","last_change":"2024-03-30T07:24:14.842838+0000","last_active":"2024-03-30T07:44:47.280896+0000","last_peered":"2024-03-30T07:44:47.280896+0000","last_clean":"2024-03-30T07:44:47.280896+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:44:47.280896+0000","last_undegraded":"2024-03-30T07:44:47.280896+0000","last_fullsized":"2024-03-30T07:44:47.280896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:31:53.743297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.989999+0000","last_change":"2024-03-30T07:18:08.005915+0000","last_active":"2024-03-30T07:44:46.989999+0000","last_peered":"2024-03-30T07:44:46.989999+0000","last_clean":"2024-03-30T07:44:46.989999+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:44:46.989999+0000","last_undegraded":"2024-03-30T07:44:46.989999+0000","last_fullsized":"2024-03-30T07:44:46.989999+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:57:08.402927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":932,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.423915+0000","last_change":"2024-03-30T07:24:15.826961+0000","last_active":"2024-03-30T07:44:48.423915+0000","last_peered":"2024-03-30T07:44:48.423915+0000","last_clean":"2024-03-30T07:44:48.423915+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:44:48.423915+0000","last_undegraded":"2024-03-30T07:44:48.423915+0000","last_fullsized":"2024-03-30T07:44:48.423915+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:17:57.852075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":909,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.615585+0000","last_change":"2024-03-30T07:18:07.657409+0000","last_active":"2024-03-30T07:44:44.615585+0000","last_peered":"2024-03-30T07:44:44.615585+0000","last_clean":"2024-03-30T07:44:44.615585+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:44:44.615585+0000","last_undegraded":"2024-03-30T07:44:44.615585+0000","last_fullsized":"2024-03-30T07:44:44.615585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:39:31.459285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":920,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.617324+0000","last_change":"2024-03-30T07:44:18.246797+0000","last_active":"2024-03-30T07:44:44.617324+0000","last_peered":"2024-03-30T07:44:44.617324+0000","last_clean":"2024-03-30T07:44:44.617324+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:44:44.617324+0000","last_undegraded":"2024-03-30T07:44:44.617324+0000","last_fullsized":"2024-03-30T07:44:44.617324+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:11.652670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.113061+0000","last_change":"2024-03-30T07:44:23.336397+0000","last_active":"2024-03-30T07:44:48.113061+0000","last_peered":"2024-03-30T07:44:48.113061+0000","last_clean":"2024-03-30T07:44:48.113061+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:44:48.113061+0000","last_undegraded":"2024-03-30T07:44:48.113061+0000","last_fullsized":"2024-03-30T07:44:48.113061+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:03:38.840636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.915968+0000","last_change":"2024-03-30T07:17:58.934819+0000","last_active":"2024-03-30T07:44:46.915968+0000","last_peered":"2024-03-30T07:44:46.915968+0000","last_clean":"2024-03-30T07:44:46.915968+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:44:46.915968+0000","last_undegraded":"2024-03-30T07:44:46.915968+0000","last_fullsized":"2024-03-30T07:44:46.915968+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:40:48.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":921,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.661867+0000","last_change":"2024-03-30T07:24:15.696366+0000","last_active":"2024-03-30T07:44:44.661867+0000","last_peered":"2024-03-30T07:44:44.661867+0000","last_clean":"2024-03-30T07:44:44.661867+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:44:44.661867+0000","last_undegraded":"2024-03-30T07:44:44.661867+0000","last_fullsized":"2024-03-30T07:44:44.661867+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:36:53.871311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.646298+0000","last_change":"2024-03-30T07:18:07.825982+0000","last_active":"2024-03-30T07:44:44.646298+0000","last_peered":"2024-03-30T07:44:44.646298+0000","last_clean":"2024-03-30T07:44:44.646298+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:44.646298+0000","last_undegraded":"2024-03-30T07:44:44.646298+0000","last_fullsized":"2024-03-30T07:44:44.646298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:07:32.853655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":924,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991550+0000","last_change":"2024-03-30T07:24:13.737991+0000","last_active":"2024-03-30T07:44:46.991550+0000","last_peered":"2024-03-30T07:44:46.991550+0000","last_clean":"2024-03-30T07:44:46.991550+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:44:46.991550+0000","last_undegraded":"2024-03-30T07:44:46.991550+0000","last_fullsized":"2024-03-30T07:44:46.991550+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:44:26.609748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.654229+0000","last_change":"2024-03-30T07:18:16.996109+0000","last_active":"2024-03-30T07:44:44.654229+0000","last_peered":"2024-03-30T07:44:44.654229+0000","last_clean":"2024-03-30T07:44:44.654229+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:44:44.654229+0000","last_undegraded":"2024-03-30T07:44:44.654229+0000","last_fullsized":"2024-03-30T07:44:44.654229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:46:51.546133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":917,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.077766+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:44:47.077766+0000","last_peered":"2024-03-30T07:44:47.077766+0000","last_clean":"2024-03-30T07:44:47.077766+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:44:47.077766+0000","last_undegraded":"2024-03-30T07:44:47.077766+0000","last_fullsized":"2024-03-30T07:44:47.077766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:41:46.497967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.970317+0000","last_change":"2024-03-30T07:18:21.573712+0000","last_active":"2024-03-30T07:44:46.970317+0000","last_peered":"2024-03-30T07:44:46.970317+0000","last_clean":"2024-03-30T07:44:46.970317+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:44:46.970317+0000","last_undegraded":"2024-03-30T07:44:46.970317+0000","last_fullsized":"2024-03-30T07:44:46.970317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:18:39.319613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":913,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.650876+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:44:44.650876+0000","last_peered":"2024-03-30T07:44:44.650876+0000","last_clean":"2024-03-30T07:44:44.650876+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:44.650876+0000","last_undegraded":"2024-03-30T07:44:44.650876+0000","last_fullsized":"2024-03-30T07:44:44.650876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.663467+0000","last_change":"2024-03-30T07:18:16.070084+0000","last_active":"2024-03-30T07:44:44.663467+0000","last_peered":"2024-03-30T07:44:44.663467+0000","last_clean":"2024-03-30T07:44:44.663467+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:44:44.663467+0000","last_undegraded":"2024-03-30T07:44:44.663467+0000","last_fullsized":"2024-03-30T07:44:44.663467+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:22.115137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.979147+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:44:46.979147+0000","last_peered":"2024-03-30T07:44:46.979147+0000","last_clean":"2024-03-30T07:44:46.979147+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:44:46.979147+0000","last_undegraded":"2024-03-30T07:44:46.979147+0000","last_fullsized":"2024-03-30T07:44:46.979147+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:12.481693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.652637+0000","last_change":"2024-03-30T07:18:17.007226+0000","last_active":"2024-03-30T07:44:44.652637+0000","last_peered":"2024-03-30T07:44:44.652637+0000","last_clean":"2024-03-30T07:44:44.652637+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:44:44.652637+0000","last_undegraded":"2024-03-30T07:44:44.652637+0000","last_fullsized":"2024-03-30T07:44:44.652637+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:02:56.775071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.340367+0000","last_change":"2024-03-30T07:44:18.307851+0000","last_active":"2024-03-30T07:44:48.340367+0000","last_peered":"2024-03-30T07:44:48.340367+0000","last_clean":"2024-03-30T07:44:48.340367+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:44:48.340367+0000","last_undegraded":"2024-03-30T07:44:48.340367+0000","last_fullsized":"2024-03-30T07:44:48.340367+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:29:33.011564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991687+0000","last_change":"2024-03-30T07:17:58.343146+0000","last_active":"2024-03-30T07:44:46.991687+0000","last_peered":"2024-03-30T07:44:46.991687+0000","last_clean":"2024-03-30T07:44:46.991687+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:44:46.991687+0000","last_undegraded":"2024-03-30T07:44:46.991687+0000","last_fullsized":"2024-03-30T07:44:46.991687+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:24:03.491281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":932,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871626+0000","last_change":"2024-03-30T07:44:23.549306+0000","last_active":"2024-03-30T07:44:46.871626+0000","last_peered":"2024-03-30T07:44:46.871626+0000","last_clean":"2024-03-30T07:44:46.871626+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:44:46.871626+0000","last_undegraded":"2024-03-30T07:44:46.871626+0000","last_fullsized":"2024-03-30T07:44:46.871626+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:34:55.114766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":927,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.059001+0000","last_change":"2024-03-30T07:44:18.161552+0000","last_active":"2024-03-30T07:44:48.059001+0000","last_peered":"2024-03-30T07:44:48.059001+0000","last_clean":"2024-03-30T07:44:48.059001+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:44:48.059001+0000","last_undegraded":"2024-03-30T07:44:48.059001+0000","last_fullsized":"2024-03-30T07:44:48.059001+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:18.359722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.884996+0000","last_change":"2024-03-30T07:24:14.186380+0000","last_active":"2024-03-30T07:44:46.884996+0000","last_peered":"2024-03-30T07:44:46.884996+0000","last_clean":"2024-03-30T07:44:46.884996+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:44:46.884996+0000","last_undegraded":"2024-03-30T07:44:46.884996+0000","last_fullsized":"2024-03-30T07:44:46.884996+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:58:59.484353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.914367+0000","last_change":"2024-03-30T07:18:07.807081+0000","last_active":"2024-03-30T07:44:46.914367+0000","last_peered":"2024-03-30T07:44:46.914367+0000","last_clean":"2024-03-30T07:44:46.914367+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:44:46.914367+0000","last_undegraded":"2024-03-30T07:44:46.914367+0000","last_fullsized":"2024-03-30T07:44:46.914367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:35:37.547969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":931,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.863858+0000","last_change":"2024-03-30T07:24:13.390417+0000","last_active":"2024-03-30T07:44:46.863858+0000","last_peered":"2024-03-30T07:44:46.863858+0000","last_clean":"2024-03-30T07:44:46.863858+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:46.863858+0000","last_undegraded":"2024-03-30T07:44:46.863858+0000","last_fullsized":"2024-03-30T07:44:46.863858+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:10:59.579650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.213477+0000","last_change":"2024-03-30T07:18:17.831393+0000","last_active":"2024-03-30T07:44:47.213477+0000","last_peered":"2024-03-30T07:44:47.213477+0000","last_clean":"2024-03-30T07:44:47.213477+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:47.213477+0000","last_undegraded":"2024-03-30T07:44:47.213477+0000","last_fullsized":"2024-03-30T07:44:47.213477+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:12.485695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.954237+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:44:46.954237+0000","last_peered":"2024-03-30T07:44:46.954237+0000","last_clean":"2024-03-30T07:44:46.954237+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:46.954237+0000","last_undegraded":"2024-03-30T07:44:46.954237+0000","last_fullsized":"2024-03-30T07:44:46.954237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:47:35.848282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1075,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.140914+0000","last_change":"2024-03-30T07:24:13.767532+0000","last_active":"2024-03-30T07:44:47.140914+0000","last_peered":"2024-03-30T07:44:47.140914+0000","last_clean":"2024-03-30T07:44:47.140914+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:47.140914+0000","last_undegraded":"2024-03-30T07:44:47.140914+0000","last_fullsized":"2024-03-30T07:44:47.140914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_clean_scrub_stamp":"2024-03-30T07:16:32.203026+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:56:57.242970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":949,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.189364+0000","last_change":"2024-03-30T07:18:08.767217+0000","last_active":"2024-03-30T07:44:47.189364+0000","last_peered":"2024-03-30T07:44:47.189364+0000","last_clean":"2024-03-30T07:44:47.189364+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:44:47.189364+0000","last_undegraded":"2024-03-30T07:44:47.189364+0000","last_fullsized":"2024-03-30T07:44:47.189364+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:53:49.994420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":955,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.223130+0000","last_change":"2024-03-30T07:24:13.784512+0000","last_active":"2024-03-30T07:44:47.223130+0000","last_peered":"2024-03-30T07:44:47.223130+0000","last_clean":"2024-03-30T07:44:47.223130+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:44:47.223130+0000","last_undegraded":"2024-03-30T07:44:47.223130+0000","last_fullsized":"2024-03-30T07:44:47.223130+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:13.663028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":926,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.114701+0000","last_change":"2024-03-30T07:44:18.218063+0000","last_active":"2024-03-30T07:44:48.114701+0000","last_peered":"2024-03-30T07:44:48.114701+0000","last_clean":"2024-03-30T07:44:48.114701+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:44:48.114701+0000","last_undegraded":"2024-03-30T07:44:48.114701+0000","last_fullsized":"2024-03-30T07:44:48.114701+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:46.659263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":910,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.930337+0000","last_change":"2024-03-30T07:18:04.168467+0000","last_active":"2024-03-30T07:44:46.930337+0000","last_peered":"2024-03-30T07:44:46.930337+0000","last_clean":"2024-03-30T07:44:46.930337+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:44:46.930337+0000","last_undegraded":"2024-03-30T07:44:46.930337+0000","last_fullsized":"2024-03-30T07:44:46.930337+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:04:44.698010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":934,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871881+0000","last_change":"2024-03-30T07:24:14.248616+0000","last_active":"2024-03-30T07:44:46.871881+0000","last_peered":"2024-03-30T07:44:46.871881+0000","last_clean":"2024-03-30T07:44:46.871881+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:44:46.871881+0000","last_undegraded":"2024-03-30T07:44:46.871881+0000","last_fullsized":"2024-03-30T07:44:46.871881+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:23:34.001344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.319814+0000","last_change":"2024-03-30T07:18:53.674429+0000","last_active":"2024-03-30T07:44:47.319814+0000","last_peered":"2024-03-30T07:44:47.319814+0000","last_clean":"2024-03-30T07:44:47.319814+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:44:47.319814+0000","last_undegraded":"2024-03-30T07:44:47.319814+0000","last_fullsized":"2024-03-30T07:44:47.319814+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:04.375300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":951,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.343877+0000","last_change":"2024-03-30T07:18:37.331986+0000","last_active":"2024-03-30T07:44:47.343877+0000","last_peered":"2024-03-30T07:44:47.343877+0000","last_clean":"2024-03-30T07:44:47.343877+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:44:47.343877+0000","last_undegraded":"2024-03-30T07:44:47.343877+0000","last_fullsized":"2024-03-30T07:44:47.343877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:54:27.069173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":905,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.647899+0000","last_change":"2024-03-30T07:18:16.744594+0000","last_active":"2024-03-30T07:44:44.647899+0000","last_peered":"2024-03-30T07:44:44.647899+0000","last_clean":"2024-03-30T07:44:44.647899+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:44:44.647899+0000","last_undegraded":"2024-03-30T07:44:44.647899+0000","last_fullsized":"2024-03-30T07:44:44.647899+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:35:24.403172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.670322+0000","last_change":"2024-03-30T07:18:16.066842+0000","last_active":"2024-03-30T07:44:44.670322+0000","last_peered":"2024-03-30T07:44:44.670322+0000","last_clean":"2024-03-30T07:44:44.670322+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:44:44.670322+0000","last_undegraded":"2024-03-30T07:44:44.670322+0000","last_fullsized":"2024-03-30T07:44:44.670322+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:16:21.059197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":912,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.892761+0000","last_change":"2024-03-30T07:18:03.197722+0000","last_active":"2024-03-30T07:44:46.892761+0000","last_peered":"2024-03-30T07:44:46.892761+0000","last_clean":"2024-03-30T07:44:46.892761+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:44:46.892761+0000","last_undegraded":"2024-03-30T07:44:46.892761+0000","last_fullsized":"2024-03-30T07:44:46.892761+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:32:02.787636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":944,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.186457+0000","last_change":"2024-03-30T07:44:18.305103+0000","last_active":"2024-03-30T07:44:47.186457+0000","last_peered":"2024-03-30T07:44:47.186457+0000","last_clean":"2024-03-30T07:44:47.186457+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:44:47.186457+0000","last_undegraded":"2024-03-30T07:44:47.186457+0000","last_fullsized":"2024-03-30T07:44:47.186457+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:16:48.555361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"43'7","reported_seq":941,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.091873+0000","last_change":"2024-03-30T07:44:18.289552+0000","last_active":"2024-03-30T07:44:48.091873+0000","last_peered":"2024-03-30T07:44:48.091873+0000","last_clean":"2024-03-30T07:44:48.091873+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:44:48.091873+0000","last_undegraded":"2024-03-30T07:44:48.091873+0000","last_fullsized":"2024-03-30T07:44:48.091873+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:06:03.894404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.376658+0000","last_change":"2024-03-30T07:24:15.802335+0000","last_active":"2024-03-30T07:44:48.376658+0000","last_peered":"2024-03-30T07:44:48.376658+0000","last_clean":"2024-03-30T07:44:48.376658+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:44:48.376658+0000","last_undegraded":"2024-03-30T07:44:48.376658+0000","last_fullsized":"2024-03-30T07:44:48.376658+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:04:34.288781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64668,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307172,"statfs":{"total":96636764160,"available":96570544128,"internally_reserved":0,"allocated":7487488,"data_stored":4185223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:25 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":18.387,"15min":18.98},"min":{"1min":5.0540000000000003,"5min":4.7649999999999997,"15min":4.7649999999999997},"max":{"1min":12.066000000000001,"5min":249.63200000000001,"15min":292.04199999999997},"last":6.2889999999999997},{"interface":"front","average":{"1min":12.994,"5min":21.666,"15min":20.100000000000001},"min":{"1min":6.0720000000000001,"5min":5.0730000000000004,"15min":4.79},"max":{"1min":39.451999999999998,"5min":248.767,"15min":299.35000000000002},"last":15.112}]},{"osd":1,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":10.112,"5min":17.547999999999998,"15min":18.082000000000001},"min":{"1min":6.3440000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":20.783999999999999,"5min":169.17099999999999,"15min":169.17099999999999},"last":7.4710000000000001},{"interface":"front","average":{"1min":12.683999999999999,"5min":24.754000000000001,"15min":25.129000000000001},"min":{"1min":8.4939999999999998,"5min":7.8929999999999998,"15min":6.7789999999999999},"max":{"1min":15.541,"5min":183.77799999999999,"15min":286.52199999999999},"last":12.396000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1180.998,"5min":530.18200000000002,"15min":228.39699999999999},"min":{"1min":9.0640000000000001,"5min":9.0640000000000001,"15min":8.0619999999999994},"max":{"1min":14020.602000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":20.991},{"interface":"front","average":{"1min":1157.28,"5min":537.84500000000003,"15min":233.36799999999999},"min":{"1min":12.426,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13714.877,"5min":13714.877,"15min":13714.877},"last":19.504000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":11.715999999999999,"5min":335.99599999999998,"15min":275.75700000000001},"min":{"1min":7.665,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":16.957999999999998,"5min":12556.546,"15min":13820.906000000001},"last":11.281000000000001},{"interface":"front","average":{"1min":14.755000000000001,"5min":341.73700000000002,"15min":281.92700000000002},"min":{"1min":9.9740000000000002,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":18.872,"5min":12557.409,"15min":13821.846},"last":15.805999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":14.395,"5min":28.911999999999999,"15min":50.296999999999997},"min":{"1min":12.161,"5min":10.385,"15min":9.8539999999999992},"max":{"1min":17.561,"5min":254.04300000000001,"15min":4768.4960000000001},"last":17.469000000000001},{"interface":"front","average":{"1min":16.638999999999999,"5min":35.829000000000001,"15min":50.470999999999997},"min":{"1min":13.890000000000001,"5min":13.061999999999999,"15min":7.8760000000000003},"max":{"1min":21.492000000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":20.231999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:42:20 2024","interfaces":[{"interface":"back","average":{"1min":57.975000000000001,"5min":40.822000000000003,"15min":28.888000000000002},"min":{"1min":14.178000000000001,"5min":12.103,"15min":6.9530000000000003},"max":{"1min":328.01600000000002,"5min":328.01600000000002,"15min":350.94400000000002},"last":14.252000000000001},{"interface":"front","average":{"1min":32.357999999999997,"5min":32.185000000000002,"15min":29.07},"min":{"1min":9.5039999999999996,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":98.340999999999994,"5min":294.49700000000001,"15min":323.77999999999997},"last":18.573}]},{"osd":7,"last update":"Sat Mar 30 07:44:07 2024","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":21.923999999999999,"15min":337.53199999999998},"min":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":19.643999999999998,"5min":334.75400000000002,"15min":19293.749},"last":12.718999999999999},{"interface":"front","average":{"1min":14.932,"5min":27.75,"15min":339.43299999999999},"min":{"1min":10.574999999999999,"5min":9.1430000000000007,"15min":8.2799999999999994},"max":{"1min":20.895,"5min":318.11200000000002,"15min":19292.978999999999},"last":11.971}]}]},{"osd":0,"up_from":22,"seq":94489280857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60540,"kb_used_data":7276,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311300,"statfs":{"total":96636764160,"available":96574771200,"internally_reserved":0,"allocated":7450624,"data_stored":4139468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":132.09700000000001},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":159.65100000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":32.520000000000003},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":36.177}]},{"osd":3,"last update":"Sat Mar 30 07:43:47 2024","interfaces":[{"interface":"back","average":{"1min":16.346,"5min":178.83799999999999,"15min":96.840000000000003},"min":{"1min":11.311,"5min":11.311,"15min":6.6429999999999998},"max":{"1min":36.655000000000001,"5min":9964.1550000000007,"15min":9964.1550000000007},"last":26.074999999999999},{"interface":"front","average":{"1min":12.576000000000001,"5min":166.869,"15min":92.548000000000002},"min":{"1min":8.0670000000000002,"5min":5.2569999999999997,"15min":5.2569999999999997},"max":{"1min":33.716000000000001,"5min":9945.7579999999998,"15min":9945.7579999999998},"last":33.866999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":168.22499999999999},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":134.59100000000001}]},{"osd":5,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":12.333,"5min":20.417999999999999,"15min":59.168999999999997},"min":{"1min":8.4209999999999994,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":17.195,"5min":298.24599999999998,"15min":7478.7280000000001},"last":35.418999999999997},{"interface":"front","average":{"1min":16.745000000000001,"5min":33.078000000000003,"15min":60.576999999999998},"min":{"1min":11.897,"5min":11.897,"15min":7.125},"max":{"1min":21.306000000000001,"5min":383.13999999999999,"15min":7477.2579999999998},"last":29.375}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":44.006999999999998,"5min":37.201999999999998,"15min":29.158999999999999},"min":{"1min":13.180999999999999,"5min":12.316000000000001,"15min":8.7569999999999997},"max":{"1min":386.452,"5min":386.452,"15min":386.452},"last":38.423000000000002},{"interface":"front","average":{"1min":31.984999999999999,"5min":28.108000000000001,"15min":25.620999999999999},"min":{"1min":8.2970000000000006,"5min":7.835,"15min":7.835},"max":{"1min":300.58199999999999,"5min":300.58199999999999,"15min":300.58199999999999},"last":37.646999999999998}]},{"osd":7,"last update":"Sat Mar 30 07:44:06 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":22.975999999999999,"15min":327.85500000000002},"min":{"1min":7.7320000000000002,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":18.007000000000001,"5min":295.38600000000002,"15min":21868.147000000001},"last":30.052},{"interface":"front","average":{"1min":13.711,"5min":24.344000000000001,"15min":331.459},"min":{"1min":10.34,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":18.896999999999998,"5min":302.11500000000001,"15min":21842.699000000001},"last":48.261000000000003}]}]},{"osd":4,"up_from":20,"seq":85899346251,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78628,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293212,"statfs":{"total":96636764160,"available":96556249088,"internally_reserved":0,"allocated":7487488,"data_stored":4198149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":6.7249999999999996,"5min":24.440999999999999,"15min":22.800999999999998},"min":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":13.228,"5min":350.36599999999999,"15min":759.322},"last":172.70699999999999},{"interface":"front","average":{"1min":8.5169999999999995,"5min":27.239000000000001,"15min":28.228000000000002},"min":{"1min":5.9379999999999997,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":11.881,"5min":410.93099999999998,"15min":761.40899999999999},"last":53.140999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":10.500999999999999,"5min":21.305,"15min":21.925000000000001},"min":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":26.32,"5min":313.54399999999998,"15min":313.54399999999998},"last":65.491},{"interface":"front","average":{"1min":13.638999999999999,"5min":29.907,"15min":29.178000000000001},"min":{"1min":8.8420000000000005,"5min":8.8420000000000005,"15min":6.3719999999999999},"max":{"1min":27.036999999999999,"5min":466.54599999999999,"15min":466.54599999999999},"last":66.787000000000006}]},{"osd":2,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":9.2159999999999993,"5min":21.456,"15min":26.949999999999999},"min":{"1min":6.3659999999999997,"5min":6.1139999999999999,"15min":5.7800000000000002},"max":{"1min":11.869,"5min":371.74200000000002,"15min":768.79100000000005},"last":33.648000000000003},{"interface":"front","average":{"1min":13.162000000000001,"5min":25.972999999999999,"15min":23.812999999999999},"min":{"1min":9.3089999999999993,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":17.018000000000001,"5min":372.322,"15min":372.322},"last":37.347000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":12.989000000000001,"5min":413.31400000000002,"15min":188.03399999999999},"min":{"1min":8.3800000000000008,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":17.893999999999998,"5min":11317.004000000001,"15min":11317.004000000001},"last":46.061999999999998},{"interface":"front","average":{"1min":15.349,"5min":405.84899999999999,"15min":187.77000000000001},"min":{"1min":8.4019999999999992,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":46.728999999999999,"5min":11222.611999999999,"15min":11222.611999999999},"last":173.566}]},{"osd":5,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":13.702,"5min":27.013000000000002,"15min":43.438000000000002},"min":{"1min":10.589,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":16.962,"5min":421.40600000000001,"15min":4807.1080000000002},"last":88.423000000000002},{"interface":"front","average":{"1min":14.109999999999999,"5min":28.047000000000001,"15min":47.658000000000001},"min":{"1min":8.6690000000000005,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":17.809999999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":73.852000000000004}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":35.414000000000001,"5min":31.486000000000001,"15min":30.466999999999999},"min":{"1min":9.8079999999999998,"5min":9.4640000000000004,"15min":9.4640000000000004},"max":{"1min":223.85499999999999,"5min":529.13699999999994,"15min":529.13699999999994},"last":72.775999999999996},{"interface":"front","average":{"1min":38.140999999999998,"5min":34.231999999999999,"15min":36.503999999999998},"min":{"1min":13.925000000000001,"5min":11.499000000000001,"15min":7.1870000000000003},"max":{"1min":223.16999999999999,"5min":529.73000000000002,"15min":967.58399999999995},"last":118.96299999999999}]},{"osd":7,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":14.84,"5min":32.366999999999997,"15min":341.73399999999998},"min":{"1min":10.010999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":19.140999999999998,"5min":530.32500000000005,"15min":20845.164000000001},"last":30.667000000000002},{"interface":"front","average":{"1min":16.094999999999999,"5min":41.709000000000003,"15min":346.267},"min":{"1min":9.3520000000000003,"5min":8.1620000000000008,"15min":7.5060000000000002},"max":{"1min":19.088000000000001,"5min":664.02300000000002,"15min":20888.814999999999},"last":29.001000000000001}]}]},{"osd":7,"up_from":19,"seq":81604378951,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60604,"kb_used_data":7336,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311236,"statfs":{"total":96636764160,"available":96574705664,"internally_reserved":0,"allocated":7512064,"data_stored":4200908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":21.870999999999999,"15min":22.189},"min":{"1min":5.0609999999999999,"5min":4.8719999999999999,"15min":4.2789999999999999},"max":{"1min":51.43,"5min":358.94499999999999,"15min":368.16899999999998},"last":12.932},{"interface":"front","average":{"1min":11.212999999999999,"5min":25.902000000000001,"15min":24.701000000000001},"min":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":50.631,"5min":349.97199999999998,"15min":349.97199999999998},"last":6.8639999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":9.4149999999999991,"5min":14.821,"15min":24.390999999999998},"min":{"1min":4.577,"5min":4.577,"15min":4.577},"max":{"1min":16.036999999999999,"5min":94.453999999999994,"15min":472.33300000000003},"last":7.5700000000000003},{"interface":"front","average":{"1min":11.449,"5min":21.518000000000001,"15min":26.239000000000001},"min":{"1min":7.4370000000000003,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":14.566000000000001,"5min":184.33500000000001,"15min":292.92099999999999},"last":12.113}]},{"osd":2,"last update":"Sat Mar 30 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":15.978999999999999,"5min":27.795000000000002,"15min":31.538},"min":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":6.6449999999999996},"max":{"1min":77.424000000000007,"5min":341.96100000000001,"15min":400.43099999999998},"last":13.747},{"interface":"front","average":{"1min":17.446000000000002,"5min":33.427,"15min":34.75},"min":{"1min":10.714,"5min":10.554,"15min":7.5049999999999999},"max":{"1min":53.476999999999997,"5min":389.37200000000001,"15min":439.31999999999999},"last":15.32}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":896.39800000000002,"5min":450.06999999999999,"15min":204.417},"min":{"1min":11.375999999999999,"5min":11.375999999999999,"15min":7.5940000000000003},"max":{"1min":13240.941000000001,"5min":13240.941000000001,"15min":13240.941000000001},"last":14.526},{"interface":"front","average":{"1min":915.66300000000001,"5min":450.815,"15min":204.88300000000001},"min":{"1min":7.484,"5min":7.423,"15min":7.423},"max":{"1min":13596.368,"5min":13596.368,"15min":13596.368},"last":9.5570000000000004}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":11.497,"5min":349.65699999999998,"15min":225.608},"min":{"1min":7.0609999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":19.181999999999999,"5min":11792.466,"15min":13877.913},"last":16.128},{"interface":"front","average":{"1min":13.494999999999999,"5min":360.642,"15min":230.58600000000001},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":19.454000000000001,"5min":11966.236999999999,"15min":14034.805},"last":8.9589999999999996}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":24.239000000000001,"15min":108.086},"min":{"1min":11.356999999999999,"5min":7.6619999999999999,"15min":7.4050000000000002},"max":{"1min":21.242000000000001,"5min":345.90499999999997,"15min":7694.1369999999997},"last":17.125},{"interface":"front","average":{"1min":12.236000000000001,"5min":22.170000000000002,"15min":112.55800000000001},"min":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":18.193000000000001,"5min":203.31299999999999,"15min":7725.4070000000002},"last":11.071}]},{"osd":6,"last update":"Sat Mar 30 07:42:11 2024","interfaces":[{"interface":"back","average":{"1min":27.544,"5min":35.518999999999998,"15min":31.760999999999999},"min":{"1min":9.8190000000000008,"5min":8.7620000000000005,"15min":8.7620000000000005},"max":{"1min":171.96100000000001,"5min":316.012,"15min":316.012},"last":18.978000000000002},{"interface":"front","average":{"1min":35.613999999999997,"5min":36.930999999999997,"15min":29.056999999999999},"min":{"1min":18.420999999999999,"5min":6.8700000000000001,"15min":6.3090000000000002},"max":{"1min":194.72399999999999,"5min":743.27700000000004,"15min":743.27700000000004},"last":18.029}]}]},{"osd":1,"up_from":16,"seq":68719477077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61016,"kb_used_data":7672,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310824,"statfs":{"total":96636764160,"available":96574283776,"internally_reserved":0,"allocated":7856128,"data_stored":4537313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":10.23,"5min":13.962,"15min":20.478000000000002},"min":{"1min":5.5880000000000001,"5min":5.2140000000000004,"15min":5.077},"max":{"1min":16.25,"5min":132.02699999999999,"15min":421.78699999999998},"last":6.0880000000000001},{"interface":"front","average":{"1min":9.8460000000000001,"5min":13.878,"15min":20.594999999999999},"min":{"1min":5.3019999999999996,"5min":5.2759999999999998,"15min":5.1139999999999999},"max":{"1min":13.212999999999999,"5min":116.801,"15min":409.91000000000003},"last":8.5020000000000007}]},{"osd":2,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":14.778,"15min":21.677},"min":{"1min":5.7489999999999997,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":18.859999999999999,"5min":180.42599999999999,"15min":433.34899999999999},"last":7.04},{"interface":"front","average":{"1min":11.426,"5min":18.071999999999999,"15min":23.861000000000001},"min":{"1min":8.8480000000000008,"5min":7.0700000000000003,"15min":5.6619999999999999},"max":{"1min":15.522,"5min":177.982,"15min":624.27300000000002},"last":15.009}]},{"osd":3,"last update":"Sat Mar 30 07:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1671.011,"5min":557.96100000000001,"15min":225.822},"min":{"1min":9.8309999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":12913.004000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":16.109999999999999},{"interface":"front","average":{"1min":1712.2249999999999,"5min":566.452,"15min":227.315},"min":{"1min":9.1899999999999995,"5min":7.1689999999999996,"15min":6.6130000000000004},"max":{"1min":13136.255999999999,"5min":13136.255999999999,"15min":13136.255999999999},"last":9.0869999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":18.109000000000002},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":9.6419999999999995}]},{"osd":5,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":17.719000000000001,"5min":20.428999999999998,"15min":47.131},"min":{"1min":12.308999999999999,"5min":9.8079999999999998,"15min":8.3439999999999994},"max":{"1min":44.511000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":17.315999999999999},{"interface":"front","average":{"1min":16.501999999999999,"5min":19.215,"15min":44.482999999999997},"min":{"1min":11.026,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":46.036000000000001,"5min":140.625,"15min":7345.2740000000003},"last":10.175000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:41:46 2024","interfaces":[{"interface":"back","average":{"1min":17.646999999999998,"5min":19.326000000000001,"15min":26.009},"min":{"1min":11.462,"5min":9.2889999999999997,"15min":7.8540000000000001},"max":{"1min":24.370000000000001,"5min":137.40199999999999,"15min":318.84100000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":12.010999999999999,"5min":13.401999999999999,"15min":24.765000000000001},"min":{"1min":8.1199999999999992,"5min":7.7110000000000003,"15min":7.4210000000000003},"max":{"1min":18.332000000000001,"5min":71.405000000000001,"15min":452.00700000000001},"last":7.774}]},{"osd":7,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":14.731999999999999,"5min":16.477,"15min":243.119},"min":{"1min":9.6280000000000001,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":20.175999999999998,"5min":115.461,"15min":20279.343000000001},"last":18.882000000000001},{"interface":"front","average":{"1min":12.869999999999999,"5min":22.413,"15min":251.726},"min":{"1min":6.4550000000000001,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":17.719999999999999,"5min":281.423,"15min":20311.815999999999},"last":10.976000000000001}]}]},{"osd":5,"up_from":231,"seq":992137445697,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":10.680999999999999},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":6.3929999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":21.510000000000002},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":11.289}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":22.34},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":15.675000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":7.6020000000000003},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":23.135999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":11.842000000000001},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":17.524000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:38 2024","interfaces":[{"interface":"back","average":{"1min":30.094999999999999,"5min":22.585000000000001,"15min":27.227},"min":{"1min":14.298999999999999,"5min":11.044,"15min":8.9009999999999998},"max":{"1min":102.39400000000001,"5min":102.39400000000001,"15min":236.04599999999999},"last":16.920000000000002},{"interface":"front","average":{"1min":21.274000000000001,"5min":17.177,"15min":24.478999999999999},"min":{"1min":10.237,"5min":9.1620000000000008,"15min":6.4249999999999998},"max":{"1min":85.435000000000002,"5min":85.435000000000002,"15min":248.31299999999999},"last":25.350000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:13 2024","interfaces":[{"interface":"back","average":{"1min":11.587,"5min":15.007,"15min":283.82900000000001},"min":{"1min":7.585,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":18.324999999999999,"5min":115.589,"15min":19723.973000000002},"last":20.396000000000001},{"interface":"front","average":{"1min":15.435,"5min":19.620999999999999,"15min":283.815},"min":{"1min":10.228999999999999,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":19.495000000000001,"5min":95.783000000000001,"15min":19724.860000000001},"last":26.346}]}]},{"osd":3,"up_from":14,"seq":60129542477,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82692,"kb_used_data":7280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289148,"statfs":{"total":96636764160,"available":96552087552,"internally_reserved":0,"allocated":7454720,"data_stored":4161164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":38.729999999999997},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":226.017}]},{"osd":1,"last update":"Sat Mar 30 07:44:28 2024","interfaces":[{"interface":"back","average":{"1min":34.786999999999999,"5min":22.965,"15min":21.632999999999999},"min":{"1min":8.6310000000000002,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":291.73599999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":227.48699999999999},{"interface":"front","average":{"1min":48.518999999999998,"5min":29.795999999999999,"15min":25.812000000000001},"min":{"1min":7.7089999999999996,"5min":7.5599999999999996,"15min":6.415},"max":{"1min":541.21199999999999,"5min":541.21199999999999,"15min":541.21199999999999},"last":159.37200000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":148.59399999999999},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":225.03899999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":158.285},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":248.155}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":231.13},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":278.62700000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:36 2024","interfaces":[{"interface":"back","average":{"1min":21.367999999999999,"5min":31.001999999999999,"15min":27.120000000000001},"min":{"1min":11.406000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":110.607,"5min":435.96600000000001,"15min":435.96600000000001},"last":172.33199999999999},{"interface":"front","average":{"1min":21.904,"5min":24.468,"15min":25.545999999999999},"min":{"1min":10.824,"5min":10.608000000000001,"15min":8.1180000000000003},"max":{"1min":62.828000000000003,"5min":145.239,"15min":259.60399999999998},"last":248.833}]},{"osd":7,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":39.734000000000002,"5min":26.948,"15min":285.68700000000001},"min":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":8.2690000000000001},"max":{"1min":527.00300000000004,"5min":527.00300000000004,"15min":20495.457999999999},"last":295.31599999999997},{"interface":"front","average":{"1min":30.413,"5min":28.954000000000001,"15min":287.50799999999998},"min":{"1min":11.025,"5min":11.025,"15min":11.025},"max":{"1min":308.75900000000001,"5min":308.75900000000001,"15min":20496.241000000002},"last":262.07799999999997}]}]},{"osd":6,"up_from":978,"seq":4200478015811,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7292,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7467008,"data_stored":4146127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":463.66800000000001,"5min":104.768,"15min":47.588000000000001},"min":{"1min":463.66800000000001,"5min":5.8390000000000004,"15min":4.6150000000000002},"max":{"1min":463.66800000000001,"5min":463.66800000000001,"15min":463.66800000000001},"last":11.76},{"interface":"front","average":{"1min":466.404,"5min":108.209,"15min":52.970999999999997},"min":{"1min":466.404,"5min":8.5449999999999999,"15min":5.8150000000000004},"max":{"1min":466.404,"5min":466.404,"15min":466.404},"last":7.5309999999999997}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":45.661999999999999,"5min":19.164999999999999,"15min":17.347000000000001},"min":{"1min":6.5149999999999997,"5min":5.4669999999999996,"15min":4.8579999999999997},"max":{"1min":298.72500000000002,"5min":298.72500000000002,"15min":298.72500000000002},"last":14.444000000000001},{"interface":"front","average":{"1min":53.417000000000002,"5min":25.199999999999999,"15min":22.443000000000001},"min":{"1min":8.4190000000000005,"5min":7.944,"15min":7.5519999999999996},"max":{"1min":323.78800000000001,"5min":323.78800000000001,"15min":323.78800000000001},"last":12.382}]},{"osd":2,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":314.89600000000002,"5min":76.939999999999998,"15min":39.972000000000001},"min":{"1min":314.89600000000002,"5min":6.351,"15min":6.351},"max":{"1min":314.89600000000002,"5min":314.89600000000002,"15min":314.89600000000002},"last":29.376999999999999},{"interface":"front","average":{"1min":353.24000000000001,"5min":87.257999999999996,"15min":46.240000000000002},"min":{"1min":353.24000000000001,"5min":8.266,"15min":8.266},"max":{"1min":353.24000000000001,"5min":353.24000000000001,"15min":353.24000000000001},"last":36.253}]},{"osd":3,"last update":"Sat Mar 30 07:42:29 2024","interfaces":[{"interface":"back","average":{"1min":22.446999999999999,"5min":188.88,"15min":130.78899999999999},"min":{"1min":9.8559999999999999,"5min":6.7830000000000004,"15min":6.7830000000000004},"max":{"1min":144.38300000000001,"5min":9266.1270000000004,"15min":9266.1270000000004},"last":8.4589999999999996},{"interface":"front","average":{"1min":16.998000000000001,"5min":177.846,"15min":123.959},"min":{"1min":7.3179999999999996,"5min":6.1200000000000001,"15min":4.8339999999999996},"max":{"1min":86.635000000000005,"5min":9082.3240000000005,"15min":9082.3240000000005},"last":18.457999999999998}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":331.70499999999998,"5min":557.46400000000006,"15min":326.00999999999999},"min":{"1min":331.70499999999998,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":331.70499999999998,"5min":13752.589,"15min":13752.589},"last":16.968},{"interface":"front","average":{"1min":469.358,"5min":575.79200000000003,"15min":334.21499999999997},"min":{"1min":469.358,"5min":7.5999999999999996,"15min":7.0199999999999996},"max":{"1min":469.358,"5min":13613.244000000001,"15min":13613.244000000001},"last":13.032999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":42.067999999999998,"5min":19.300999999999998,"15min":110.551},"min":{"1min":8.1010000000000009,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":483.904,"5min":483.904,"15min":9617.3179999999993},"last":16.385999999999999},{"interface":"front","average":{"1min":46.238,"5min":24.486000000000001,"15min":110.789},"min":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":8.8580000000000005},"max":{"1min":485.661,"5min":485.661,"15min":9386.0400000000009},"last":9.8219999999999992}]},{"osd":7,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":67.293000000000006,"5min":34.445,"15min":29.379999999999999},"min":{"1min":12.813000000000001,"5min":7.9729999999999999,"15min":7.9729999999999999},"max":{"1min":470.34300000000002,"5min":470.34300000000002,"15min":470.34300000000002},"last":17.635000000000002},{"interface":"front","average":{"1min":70.335999999999999,"5min":35.420000000000002,"15min":29.364000000000001},"min":{"1min":12.48,"5min":8.452,"15min":7.4189999999999996},"max":{"1min":487.30700000000002,"5min":487.30700000000002,"15min":487.30700000000002},"last":15.579000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:44:51.080 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T07:44:51.080 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.178+0000 7f032603a640 10 monclient: get_monmap_and_config 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.178+0000 7f032603a640 10 monclient: build_initial_monmap 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.178+0000 7f032603a640 10 monclient: monmap: 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:51.179653+0000 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:51.179653+0000 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:51.181 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:51.182 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:51.182 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.179+0000 7f032603a640 1 Processor -- start 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.179+0000 7f032603a640 1 -- start start 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.179+0000 7f032603a640 10 monclient: init 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-03-30T07:44:51.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0320071be0 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f03200b0ca0 0x7f03200a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): picked mon.noname-c con 0x7f03200b0ca0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f032013e820 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200a8cc0 con 0x7f0320071be0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200aa380 con 0x7f032013e820 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0320071fc0 con 0x7f03200b0ca0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f032603a640 10 monclient(hunting): authenticate will time out at 3217.295456s 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f031f7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0320071be0 0x7f0320144a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55214/0 (socket says 172.21.15.47:55214) 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.180+0000 7f031f7fe640 1 -- 172.21.15.47:0/1553497218 learned_addr learned my addr 172.21.15.47:0/1553497218 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.181+0000 7f031effd640 1 --2- 172.21.15.47:0/1553497218 >> v2:172.21.15.52:3300/0 conn(0x7f03200b0ca0 0x7f03200a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.181+0000 7f031effd640 10 monclient(hunting): get_auth_request con 0x7f03200b0ca0 auth_method 0 2024-03-30T07:44:51.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.181+0000 7f031effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.181+0000 7f031effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.181+0000 7f031effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.182+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.182+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.182+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 1 -- 172.21.15.47:0/1553497218 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233512967 0 0) 0x7f03200a8cc0 con 0x7f0320071be0 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 10 monclient(hunting): _negotiate 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 10 monclient(hunting): my global_id is 15822 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.183+0000 7f031e7fc640 1 -- 172.21.15.47:0/1553497218 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0308003520 con 0x7f0320071be0 2024-03-30T07:44:51.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 10 monclient(hunting): handle_auth_done global_id 22318 payload 274 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 1 -- 172.21.15.47:0/1553497218 >> v1:172.21.15.52:6789/0 conn(0x7f032013e820 legacy=0x7f032014bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 1 -- 172.21.15.47:0/1553497218 >> v1:172.21.15.47:6789/0 conn(0x7f0320071be0 legacy=0x7f0320144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 10 monclient: _finish_hunting 0 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 1 monclient: found mon.noname-c 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.184+0000 7f031effd640 1 -- 172.21.15.47:0/1553497218 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200a8970 con 0x7f03200b0ca0 2024-03-30T07:44:51.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.187+0000 7f031effd640 1 --2- 172.21.15.47:0/1553497218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03200b0ca0 0x7f03200a9150 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f03140097a0 tx=0x7f031402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed71055d786e2b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 1 -- 172.21.15.47:0/1553497218 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314030610 con 0x7f03200b0ca0 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: dump: 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.193 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 1 -- 172.21.15.47:0/1553497218 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0314030770 con 0x7f03200b0ca0 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 1 -- 172.21.15.47:0/1553497218 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314030b00 con 0x7f03200b0ca0 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f032603a640 5 monclient: authenticate success, global_id 22318 2024-03-30T07:44:51.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f032603a640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f032603a640 4 set_mon_vals no callback set 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.191+0000 7f031e7fc640 10 monclient: dump: 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.195 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 10 monclient: shutdown 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 -- 172.21.15.47:0/1553497218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03200b0ca0 msgr2=0x7f03200a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 --2- 172.21.15.47:0/1553497218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03200b0ca0 0x7f03200a9150 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f03140097a0 tx=0x7f031402b960 comp rx=0 tx=0).stop 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 -- 172.21.15.47:0/1553497218 shutdown_connections 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 --2- 172.21.15.47:0/1553497218 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f03200b0ca0 0x7f03200a9150 unknown :-1 s=CLOSED pgs=5506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 -- 172.21.15.47:0/1553497218 >> 172.21.15.47:0/1553497218 conn(0x7f03200b73d0 msgr2=0x7f03200b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 -- 172.21.15.47:0/1553497218 shutdown_connections 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 1 -- 172.21.15.47:0/1553497218 wait complete. 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 10 monclient: build_initial_monmap 2024-03-30T07:44:51.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.192+0000 7f032603a640 10 monclient: monmap: 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 librados: starting msgr at 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 librados: starting objecter 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 Processor -- start 2024-03-30T07:44:51.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 -- start start 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 librados: setting wanted keys 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 1 librados: calling monclient init 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 10 monclient: init 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.193+0000 7f032603a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0320071be0 0x7f0320134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): picked mon.noname-a con 0x7f0320071be0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 0x7f03201d63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): picked mon.noname-c con 0x7f03200b0ca0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 0x7f03201da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): picked mon.noname-b con 0x7f032013e820 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03201d9d60 con 0x7f0320071be0 2024-03-30T07:44:51.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03201d68f0 con 0x7f032013e820 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03201ca070 con 0x7f03200b0ca0 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f032603a640 10 monclient(hunting): authenticate will time out at 3217.309220s 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031f7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0320071be0 0x7f0320134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 0x7f03201d63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031ffff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 0x7f03201da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 0x7f03201d63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:59960/0 (socket says 172.21.15.47:59960) 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 1 -- 172.21.15.47:0/912180466 learned_addr learned my addr 172.21.15.47:0/912180466 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 10 monclient(hunting): get_auth_request con 0x7f03200b0ca0 auth_method 0 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031f7fe640 10 monclient(hunting): get_auth_request con 0x7f0320071be0 auth_method 0 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.194+0000 7f031f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): get_auth_request con 0x7f032013e820 auth_method 0 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.195+0000 7f031ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 10 monclient(hunting): handle_auth_done global_id 22321 payload 824 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 -- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 msgr2=0x7f03201d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 0x7f03201d63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 -- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0320071be0 msgr2=0x7f0320134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0320071be0 0x7f0320134bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 monclient: found mon.noname-b 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:51.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.198+0000 7f031ffff640 1 -- 172.21.15.47:0/912180466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314009400 con 0x7f032013e820 2024-03-30T07:44:51.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.200+0000 7f031ffff640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 0x7f03201da1f0 secure :-1 s=READY pgs=5507 cs=0 l=1 rev1=1 crypto rx=0x7f031800c700 tx=0x7f031800cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.200+0000 7f0307fff640 10 client.?.objecter ms_handle_connect 0x7f032013e820 2024-03-30T07:44:51.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.200+0000 7f0307fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318003b70 con 0x7f032013e820 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: dump: 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.207 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: _finish_auth 0 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: _check_auth_tickets 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0318003cd0 con 0x7f032013e820 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318005310 con 0x7f032013e820 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f032603a640 5 monclient: authenticate success, global_id 22321 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f032603a640 10 monclient: _renew_subs 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f032603a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320143bd0 con 0x7f032013e820 2024-03-30T07:44:51.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f0307fff640 10 monclient: dump: 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.203+0000 7f031dffb640 4 set_mon_vals no callback set 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.204+0000 7f032603a640 10 client.22321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.204+0000 7f032603a640 10 monclient: _renew_subs 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.204+0000 7f032603a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.204+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0320144780 con 0x7f032013e820 2024-03-30T07:44:51.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.204+0000 7f032603a640 1 librados: init done 2024-03-30T07:44:51.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.205+0000 7f032603a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:51.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.205+0000 7f032603a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:51.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.205+0000 7f032603a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.205+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03201d9ed0 con 0x7f032013e820 2024-03-30T07:44:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.223+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0318005470 con 0x7f032013e820 2024-03-30T07:44:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.223+0000 7f0307fff640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 0x7f030c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.223+0000 7f031f7fe640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 0x7f030c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.223+0000 7f031f7fe640 10 monclient: get_auth_request con 0x7f030c06c5e0 auth_method 0 2024-03-30T07:44:51.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.224+0000 7f031f7fe640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 0x7f030c06eaa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03100011e0 tx=0x7f0310007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.224+0000 7f0307fff640 10 client.22321.objecter ms_handle_connect 0x7f030c06c5e0 2024-03-30T07:44:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.225+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 473..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f03180588c0 con 0x7f032013e820 2024-03-30T07:44:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.225+0000 7f0307fff640 10 client.22321.objecter ms_dispatch 0x7f0320000b90 osd_map(994..994 src has 473..994) v4 2024-03-30T07:44:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.225+0000 7f0307fff640 3 client.22321.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.225+0000 7f0307fff640 3 client.22321.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.225+0000 7f0307fff640 20 client.22321.objecter dump_active .. 0 homeless 2024-03-30T07:44:51.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.482+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0318054c00 con 0x7f032013e820 2024-03-30T07:44:51.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.482+0000 7f0307fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:51.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.482+0000 7f0307fff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:51.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.661+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f03200b37b0 con 0x7f030c06c5e0 2024-03-30T07:44:51.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.666+0000 7f0307fff640 1 -- 172.21.15.47:0/912180466 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152209 (secure 0 0 0) 0x7f03100014d0 con 0x7f030c06c5e0 2024-03-30T07:44:51.670 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:44:51.670 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.668+0000 7f032603a640 10 librados: watch_flush enter 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.668+0000 7f032603a640 10 librados: watch_flush exit 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.668+0000 7f032603a640 20 client.22321.objecter shutdown clearing up homeless session... 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.668+0000 7f032603a640 10 client.22321.objecter successfully canceled tick 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 msgr2=0x7f030c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 0x7f030c06eaa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03100011e0 tx=0x7f0310007d30 comp rx=0 tx=0).stop 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 10 monclient: shutdown 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:51.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 msgr2=0x7f03201da1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 0x7f03201da1f0 secure :-1 s=READY pgs=5507 cs=0 l=1 rev1=1 crypto rx=0x7f031800c700 tx=0x7f031800cbc0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 shutdown_connections 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f030c06c5e0 0x7f030c06eaa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f032013e820 0x7f03201da1f0 unknown :-1 s=CLOSED pgs=5507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03200b0ca0 0x7f03201d63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 --2- 172.21.15.47:0/912180466 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0320071be0 0x7f0320134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 >> 172.21.15.47:0/912180466 conn(0x7f03200b73d0 msgr2=0x7f032013ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 shutdown_connections 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 -- 172.21.15.47:0/912180466 wait complete. 2024-03-30T07:44:51.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.669+0000 7f032603a640 1 librados: shutdown 2024-03-30T07:44:51.681 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1848,"stamp":"2024-03-30T07:44:50.371781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":149,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529660,"kb_used_data":59172,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445060,"statfs":{"total":773094113280,"available":772551741440,"internally_reserved":0,"allocated":60592128,"data_stored":34119774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":625,"apply_latency_ms":625,"commit_latency_ns":625000000,"apply_latency_ns":625000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.254424+0000","last_change":"2024-03-30T07:18:12.759583+0000","last_active":"2024-03-30T07:44:47.254424+0000","last_peered":"2024-03-30T07:44:47.254424+0000","last_clean":"2024-03-30T07:44:47.254424+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:44:47.254424+0000","last_undegraded":"2024-03-30T07:44:47.254424+0000","last_fullsized":"2024-03-30T07:44:47.254424+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:38:53.027870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.280896+0000","last_change":"2024-03-30T07:24:14.842838+0000","last_active":"2024-03-30T07:44:47.280896+0000","last_peered":"2024-03-30T07:44:47.280896+0000","last_clean":"2024-03-30T07:44:47.280896+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:44:47.280896+0000","last_undegraded":"2024-03-30T07:44:47.280896+0000","last_fullsized":"2024-03-30T07:44:47.280896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:31:53.743297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.989999+0000","last_change":"2024-03-30T07:18:08.005915+0000","last_active":"2024-03-30T07:44:46.989999+0000","last_peered":"2024-03-30T07:44:46.989999+0000","last_clean":"2024-03-30T07:44:46.989999+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:44:46.989999+0000","last_undegraded":"2024-03-30T07:44:46.989999+0000","last_fullsized":"2024-03-30T07:44:46.989999+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:57:08.402927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":932,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.423915+0000","last_change":"2024-03-30T07:24:15.826961+0000","last_active":"2024-03-30T07:44:48.423915+0000","last_peered":"2024-03-30T07:44:48.423915+0000","last_clean":"2024-03-30T07:44:48.423915+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:44:48.423915+0000","last_undegraded":"2024-03-30T07:44:48.423915+0000","last_fullsized":"2024-03-30T07:44:48.423915+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:17:57.852075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":909,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.615585+0000","last_change":"2024-03-30T07:18:07.657409+0000","last_active":"2024-03-30T07:44:44.615585+0000","last_peered":"2024-03-30T07:44:44.615585+0000","last_clean":"2024-03-30T07:44:44.615585+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:44:44.615585+0000","last_undegraded":"2024-03-30T07:44:44.615585+0000","last_fullsized":"2024-03-30T07:44:44.615585+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:39:31.459285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":920,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.617324+0000","last_change":"2024-03-30T07:44:18.246797+0000","last_active":"2024-03-30T07:44:44.617324+0000","last_peered":"2024-03-30T07:44:44.617324+0000","last_clean":"2024-03-30T07:44:44.617324+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:44:44.617324+0000","last_undegraded":"2024-03-30T07:44:44.617324+0000","last_fullsized":"2024-03-30T07:44:44.617324+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:11.652670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.113061+0000","last_change":"2024-03-30T07:44:23.336397+0000","last_active":"2024-03-30T07:44:48.113061+0000","last_peered":"2024-03-30T07:44:48.113061+0000","last_clean":"2024-03-30T07:44:48.113061+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:44:48.113061+0000","last_undegraded":"2024-03-30T07:44:48.113061+0000","last_fullsized":"2024-03-30T07:44:48.113061+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:03:38.840636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.915968+0000","last_change":"2024-03-30T07:17:58.934819+0000","last_active":"2024-03-30T07:44:46.915968+0000","last_peered":"2024-03-30T07:44:46.915968+0000","last_clean":"2024-03-30T07:44:46.915968+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:44:46.915968+0000","last_undegraded":"2024-03-30T07:44:46.915968+0000","last_fullsized":"2024-03-30T07:44:46.915968+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:40:48.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":921,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.661867+0000","last_change":"2024-03-30T07:24:15.696366+0000","last_active":"2024-03-30T07:44:44.661867+0000","last_peered":"2024-03-30T07:44:44.661867+0000","last_clean":"2024-03-30T07:44:44.661867+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:44:44.661867+0000","last_undegraded":"2024-03-30T07:44:44.661867+0000","last_fullsized":"2024-03-30T07:44:44.661867+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:36:53.871311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.646298+0000","last_change":"2024-03-30T07:18:07.825982+0000","last_active":"2024-03-30T07:44:44.646298+0000","last_peered":"2024-03-30T07:44:44.646298+0000","last_clean":"2024-03-30T07:44:44.646298+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:44.646298+0000","last_undegraded":"2024-03-30T07:44:44.646298+0000","last_fullsized":"2024-03-30T07:44:44.646298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:07:32.853655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":924,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991550+0000","last_change":"2024-03-30T07:24:13.737991+0000","last_active":"2024-03-30T07:44:46.991550+0000","last_peered":"2024-03-30T07:44:46.991550+0000","last_clean":"2024-03-30T07:44:46.991550+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:44:46.991550+0000","last_undegraded":"2024-03-30T07:44:46.991550+0000","last_fullsized":"2024-03-30T07:44:46.991550+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:44:26.609748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.654229+0000","last_change":"2024-03-30T07:18:16.996109+0000","last_active":"2024-03-30T07:44:44.654229+0000","last_peered":"2024-03-30T07:44:44.654229+0000","last_clean":"2024-03-30T07:44:44.654229+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:44:44.654229+0000","last_undegraded":"2024-03-30T07:44:44.654229+0000","last_fullsized":"2024-03-30T07:44:44.654229+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:46:51.546133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":917,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.077766+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:44:47.077766+0000","last_peered":"2024-03-30T07:44:47.077766+0000","last_clean":"2024-03-30T07:44:47.077766+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:44:47.077766+0000","last_undegraded":"2024-03-30T07:44:47.077766+0000","last_fullsized":"2024-03-30T07:44:47.077766+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:41:46.497967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.970317+0000","last_change":"2024-03-30T07:18:21.573712+0000","last_active":"2024-03-30T07:44:46.970317+0000","last_peered":"2024-03-30T07:44:46.970317+0000","last_clean":"2024-03-30T07:44:46.970317+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:44:46.970317+0000","last_undegraded":"2024-03-30T07:44:46.970317+0000","last_fullsized":"2024-03-30T07:44:46.970317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:18:39.319613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":913,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.650876+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:44:44.650876+0000","last_peered":"2024-03-30T07:44:44.650876+0000","last_clean":"2024-03-30T07:44:44.650876+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:44.650876+0000","last_undegraded":"2024-03-30T07:44:44.650876+0000","last_fullsized":"2024-03-30T07:44:44.650876+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":907,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.663467+0000","last_change":"2024-03-30T07:18:16.070084+0000","last_active":"2024-03-30T07:44:44.663467+0000","last_peered":"2024-03-30T07:44:44.663467+0000","last_clean":"2024-03-30T07:44:44.663467+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:44:44.663467+0000","last_undegraded":"2024-03-30T07:44:44.663467+0000","last_fullsized":"2024-03-30T07:44:44.663467+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:22.115137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.979147+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:44:46.979147+0000","last_peered":"2024-03-30T07:44:46.979147+0000","last_clean":"2024-03-30T07:44:46.979147+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:44:46.979147+0000","last_undegraded":"2024-03-30T07:44:46.979147+0000","last_fullsized":"2024-03-30T07:44:46.979147+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:12.481693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.652637+0000","last_change":"2024-03-30T07:18:17.007226+0000","last_active":"2024-03-30T07:44:44.652637+0000","last_peered":"2024-03-30T07:44:44.652637+0000","last_clean":"2024-03-30T07:44:44.652637+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:44:44.652637+0000","last_undegraded":"2024-03-30T07:44:44.652637+0000","last_fullsized":"2024-03-30T07:44:44.652637+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:02:56.775071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.340367+0000","last_change":"2024-03-30T07:44:18.307851+0000","last_active":"2024-03-30T07:44:48.340367+0000","last_peered":"2024-03-30T07:44:48.340367+0000","last_clean":"2024-03-30T07:44:48.340367+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:44:48.340367+0000","last_undegraded":"2024-03-30T07:44:48.340367+0000","last_fullsized":"2024-03-30T07:44:48.340367+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:29:33.011564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.991687+0000","last_change":"2024-03-30T07:17:58.343146+0000","last_active":"2024-03-30T07:44:46.991687+0000","last_peered":"2024-03-30T07:44:46.991687+0000","last_clean":"2024-03-30T07:44:46.991687+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:44:46.991687+0000","last_undegraded":"2024-03-30T07:44:46.991687+0000","last_fullsized":"2024-03-30T07:44:46.991687+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:24:03.491281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":932,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871626+0000","last_change":"2024-03-30T07:44:23.549306+0000","last_active":"2024-03-30T07:44:46.871626+0000","last_peered":"2024-03-30T07:44:46.871626+0000","last_clean":"2024-03-30T07:44:46.871626+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:44:46.871626+0000","last_undegraded":"2024-03-30T07:44:46.871626+0000","last_fullsized":"2024-03-30T07:44:46.871626+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:34:55.114766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":927,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.059001+0000","last_change":"2024-03-30T07:44:18.161552+0000","last_active":"2024-03-30T07:44:48.059001+0000","last_peered":"2024-03-30T07:44:48.059001+0000","last_clean":"2024-03-30T07:44:48.059001+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:44:48.059001+0000","last_undegraded":"2024-03-30T07:44:48.059001+0000","last_fullsized":"2024-03-30T07:44:48.059001+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:18.359722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.884996+0000","last_change":"2024-03-30T07:24:14.186380+0000","last_active":"2024-03-30T07:44:46.884996+0000","last_peered":"2024-03-30T07:44:46.884996+0000","last_clean":"2024-03-30T07:44:46.884996+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:44:46.884996+0000","last_undegraded":"2024-03-30T07:44:46.884996+0000","last_fullsized":"2024-03-30T07:44:46.884996+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:58:59.484353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":920,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.914367+0000","last_change":"2024-03-30T07:18:07.807081+0000","last_active":"2024-03-30T07:44:46.914367+0000","last_peered":"2024-03-30T07:44:46.914367+0000","last_clean":"2024-03-30T07:44:46.914367+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:44:46.914367+0000","last_undegraded":"2024-03-30T07:44:46.914367+0000","last_fullsized":"2024-03-30T07:44:46.914367+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:35:37.547969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":931,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.863858+0000","last_change":"2024-03-30T07:24:13.390417+0000","last_active":"2024-03-30T07:44:46.863858+0000","last_peered":"2024-03-30T07:44:46.863858+0000","last_clean":"2024-03-30T07:44:46.863858+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:46.863858+0000","last_undegraded":"2024-03-30T07:44:46.863858+0000","last_fullsized":"2024-03-30T07:44:46.863858+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:10:59.579650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":943,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.213477+0000","last_change":"2024-03-30T07:18:17.831393+0000","last_active":"2024-03-30T07:44:47.213477+0000","last_peered":"2024-03-30T07:44:47.213477+0000","last_clean":"2024-03-30T07:44:47.213477+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:47.213477+0000","last_undegraded":"2024-03-30T07:44:47.213477+0000","last_fullsized":"2024-03-30T07:44:47.213477+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:12.485695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.954237+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:44:46.954237+0000","last_peered":"2024-03-30T07:44:46.954237+0000","last_clean":"2024-03-30T07:44:46.954237+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:46.954237+0000","last_undegraded":"2024-03-30T07:44:46.954237+0000","last_fullsized":"2024-03-30T07:44:46.954237+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:47:35.848282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'76","reported_seq":1075,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.140914+0000","last_change":"2024-03-30T07:24:13.767532+0000","last_active":"2024-03-30T07:44:47.140914+0000","last_peered":"2024-03-30T07:44:47.140914+0000","last_clean":"2024-03-30T07:44:47.140914+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:47.140914+0000","last_undegraded":"2024-03-30T07:44:47.140914+0000","last_fullsized":"2024-03-30T07:44:47.140914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:32.203026+0000","last_clean_scrub_stamp":"2024-03-30T07:16:32.203026+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:56:57.242970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":949,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.189364+0000","last_change":"2024-03-30T07:18:08.767217+0000","last_active":"2024-03-30T07:44:47.189364+0000","last_peered":"2024-03-30T07:44:47.189364+0000","last_clean":"2024-03-30T07:44:47.189364+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:44:47.189364+0000","last_undegraded":"2024-03-30T07:44:47.189364+0000","last_fullsized":"2024-03-30T07:44:47.189364+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:53:49.994420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":955,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.223130+0000","last_change":"2024-03-30T07:24:13.784512+0000","last_active":"2024-03-30T07:44:47.223130+0000","last_peered":"2024-03-30T07:44:47.223130+0000","last_clean":"2024-03-30T07:44:47.223130+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:44:47.223130+0000","last_undegraded":"2024-03-30T07:44:47.223130+0000","last_fullsized":"2024-03-30T07:44:47.223130+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:54:13.663028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":926,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.114701+0000","last_change":"2024-03-30T07:44:18.218063+0000","last_active":"2024-03-30T07:44:48.114701+0000","last_peered":"2024-03-30T07:44:48.114701+0000","last_clean":"2024-03-30T07:44:48.114701+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:44:48.114701+0000","last_undegraded":"2024-03-30T07:44:48.114701+0000","last_fullsized":"2024-03-30T07:44:48.114701+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:46.659263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":910,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.930337+0000","last_change":"2024-03-30T07:18:04.168467+0000","last_active":"2024-03-30T07:44:46.930337+0000","last_peered":"2024-03-30T07:44:46.930337+0000","last_clean":"2024-03-30T07:44:46.930337+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:44:46.930337+0000","last_undegraded":"2024-03-30T07:44:46.930337+0000","last_fullsized":"2024-03-30T07:44:46.930337+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:04:44.698010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":934,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.871881+0000","last_change":"2024-03-30T07:24:14.248616+0000","last_active":"2024-03-30T07:44:46.871881+0000","last_peered":"2024-03-30T07:44:46.871881+0000","last_clean":"2024-03-30T07:44:46.871881+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:44:46.871881+0000","last_undegraded":"2024-03-30T07:44:46.871881+0000","last_fullsized":"2024-03-30T07:44:46.871881+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:23:34.001344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":930,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.319814+0000","last_change":"2024-03-30T07:18:53.674429+0000","last_active":"2024-03-30T07:44:47.319814+0000","last_peered":"2024-03-30T07:44:47.319814+0000","last_clean":"2024-03-30T07:44:47.319814+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:44:47.319814+0000","last_undegraded":"2024-03-30T07:44:47.319814+0000","last_fullsized":"2024-03-30T07:44:47.319814+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:04.375300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":951,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.343877+0000","last_change":"2024-03-30T07:18:37.331986+0000","last_active":"2024-03-30T07:44:47.343877+0000","last_peered":"2024-03-30T07:44:47.343877+0000","last_clean":"2024-03-30T07:44:47.343877+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:44:47.343877+0000","last_undegraded":"2024-03-30T07:44:47.343877+0000","last_fullsized":"2024-03-30T07:44:47.343877+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:54:27.069173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":905,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.647899+0000","last_change":"2024-03-30T07:18:16.744594+0000","last_active":"2024-03-30T07:44:44.647899+0000","last_peered":"2024-03-30T07:44:44.647899+0000","last_clean":"2024-03-30T07:44:44.647899+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:44:44.647899+0000","last_undegraded":"2024-03-30T07:44:44.647899+0000","last_fullsized":"2024-03-30T07:44:44.647899+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:35:24.403172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":906,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-30T07:44:44.670322+0000","last_change":"2024-03-30T07:18:16.066842+0000","last_active":"2024-03-30T07:44:44.670322+0000","last_peered":"2024-03-30T07:44:44.670322+0000","last_clean":"2024-03-30T07:44:44.670322+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:44:44.670322+0000","last_undegraded":"2024-03-30T07:44:44.670322+0000","last_fullsized":"2024-03-30T07:44:44.670322+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:16:21.059197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":912,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:46.892761+0000","last_change":"2024-03-30T07:18:03.197722+0000","last_active":"2024-03-30T07:44:46.892761+0000","last_peered":"2024-03-30T07:44:46.892761+0000","last_clean":"2024-03-30T07:44:46.892761+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:44:46.892761+0000","last_undegraded":"2024-03-30T07:44:46.892761+0000","last_fullsized":"2024-03-30T07:44:46.892761+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:32:02.787636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":944,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-03-30T07:44:47.186457+0000","last_change":"2024-03-30T07:44:18.305103+0000","last_active":"2024-03-30T07:44:47.186457+0000","last_peered":"2024-03-30T07:44:47.186457+0000","last_clean":"2024-03-30T07:44:47.186457+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:44:47.186457+0000","last_undegraded":"2024-03-30T07:44:47.186457+0000","last_fullsized":"2024-03-30T07:44:47.186457+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:16:48.555361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"43'7","reported_seq":941,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.091873+0000","last_change":"2024-03-30T07:44:18.289552+0000","last_active":"2024-03-30T07:44:48.091873+0000","last_peered":"2024-03-30T07:44:48.091873+0000","last_clean":"2024-03-30T07:44:48.091873+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:44:48.091873+0000","last_undegraded":"2024-03-30T07:44:48.091873+0000","last_fullsized":"2024-03-30T07:44:48.091873+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:06:03.894404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.376658+0000","last_change":"2024-03-30T07:24:15.802335+0000","last_active":"2024-03-30T07:44:48.376658+0000","last_peered":"2024-03-30T07:44:48.376658+0000","last_clean":"2024-03-30T07:44:48.376658+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:44:48.376658+0000","last_undegraded":"2024-03-30T07:44:48.376658+0000","last_fullsized":"2024-03-30T07:44:48.376658+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:04:34.288781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64668,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307172,"statfs":{"total":96636764160,"available":96570544128,"internally_reserved":0,"allocated":7487488,"data_stored":4185223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":145,"apply_latency_ms":145,"commit_latency_ns":145000000,"apply_latency_ns":145000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:25 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":18.387,"15min":18.98},"min":{"1min":5.0540000000000003,"5min":4.7649999999999997,"15min":4.7649999999999997},"max":{"1min":12.066000000000001,"5min":249.63200000000001,"15min":292.04199999999997},"last":6.2889999999999997},{"interface":"front","average":{"1min":12.994,"5min":21.666,"15min":20.100000000000001},"min":{"1min":6.0720000000000001,"5min":5.0730000000000004,"15min":4.79},"max":{"1min":39.451999999999998,"5min":248.767,"15min":299.35000000000002},"last":15.112}]},{"osd":1,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":10.112,"5min":17.547999999999998,"15min":18.082000000000001},"min":{"1min":6.3440000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":20.783999999999999,"5min":169.17099999999999,"15min":169.17099999999999},"last":7.4710000000000001},{"interface":"front","average":{"1min":12.683999999999999,"5min":24.754000000000001,"15min":25.129000000000001},"min":{"1min":8.4939999999999998,"5min":7.8929999999999998,"15min":6.7789999999999999},"max":{"1min":15.541,"5min":183.77799999999999,"15min":286.52199999999999},"last":12.396000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1180.998,"5min":530.18200000000002,"15min":228.39699999999999},"min":{"1min":9.0640000000000001,"5min":9.0640000000000001,"15min":8.0619999999999994},"max":{"1min":14020.602000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":20.991},{"interface":"front","average":{"1min":1157.28,"5min":537.84500000000003,"15min":233.36799999999999},"min":{"1min":12.426,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13714.877,"5min":13714.877,"15min":13714.877},"last":19.504000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":11.715999999999999,"5min":335.99599999999998,"15min":275.75700000000001},"min":{"1min":7.665,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":16.957999999999998,"5min":12556.546,"15min":13820.906000000001},"last":11.281000000000001},{"interface":"front","average":{"1min":14.755000000000001,"5min":341.73700000000002,"15min":281.92700000000002},"min":{"1min":9.9740000000000002,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":18.872,"5min":12557.409,"15min":13821.846},"last":15.805999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:43:54 2024","interfaces":[{"interface":"back","average":{"1min":14.395,"5min":28.911999999999999,"15min":50.296999999999997},"min":{"1min":12.161,"5min":10.385,"15min":9.8539999999999992},"max":{"1min":17.561,"5min":254.04300000000001,"15min":4768.4960000000001},"last":17.469000000000001},{"interface":"front","average":{"1min":16.638999999999999,"5min":35.829000000000001,"15min":50.470999999999997},"min":{"1min":13.890000000000001,"5min":13.061999999999999,"15min":7.8760000000000003},"max":{"1min":21.492000000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":20.231999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:42:20 2024","interfaces":[{"interface":"back","average":{"1min":57.975000000000001,"5min":40.822000000000003,"15min":28.888000000000002},"min":{"1min":14.178000000000001,"5min":12.103,"15min":6.9530000000000003},"max":{"1min":328.01600000000002,"5min":328.01600000000002,"15min":350.94400000000002},"last":14.252000000000001},{"interface":"front","average":{"1min":32.357999999999997,"5min":32.185000000000002,"15min":29.07},"min":{"1min":9.5039999999999996,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":98.340999999999994,"5min":294.49700000000001,"15min":323.77999999999997},"last":18.573}]},{"osd":7,"last update":"Sat Mar 30 07:44:07 2024","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":21.923999999999999,"15min":337.53199999999998},"min":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":19.643999999999998,"5min":334.75400000000002,"15min":19293.749},"last":12.718999999999999},{"interface":"front","average":{"1min":14.932,"5min":27.75,"15min":339.43299999999999},"min":{"1min":10.574999999999999,"5min":9.1430000000000007,"15min":8.2799999999999994},"max":{"1min":20.895,"5min":318.11200000000002,"15min":19292.978999999999},"last":11.971}]}]},{"osd":0,"up_from":22,"seq":94489280857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60540,"kb_used_data":7276,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311300,"statfs":{"total":96636764160,"available":96574771200,"internally_reserved":0,"allocated":7450624,"data_stored":4139468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":92,"apply_latency_ms":92,"commit_latency_ns":92000000,"apply_latency_ns":92000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":132.09700000000001},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":159.65100000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":32.520000000000003},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":36.177}]},{"osd":3,"last update":"Sat Mar 30 07:43:47 2024","interfaces":[{"interface":"back","average":{"1min":16.346,"5min":178.83799999999999,"15min":96.840000000000003},"min":{"1min":11.311,"5min":11.311,"15min":6.6429999999999998},"max":{"1min":36.655000000000001,"5min":9964.1550000000007,"15min":9964.1550000000007},"last":26.074999999999999},{"interface":"front","average":{"1min":12.576000000000001,"5min":166.869,"15min":92.548000000000002},"min":{"1min":8.0670000000000002,"5min":5.2569999999999997,"15min":5.2569999999999997},"max":{"1min":33.716000000000001,"5min":9945.7579999999998,"15min":9945.7579999999998},"last":33.866999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":168.22499999999999},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":134.59100000000001}]},{"osd":5,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":12.333,"5min":20.417999999999999,"15min":59.168999999999997},"min":{"1min":8.4209999999999994,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":17.195,"5min":298.24599999999998,"15min":7478.7280000000001},"last":35.418999999999997},{"interface":"front","average":{"1min":16.745000000000001,"5min":33.078000000000003,"15min":60.576999999999998},"min":{"1min":11.897,"5min":11.897,"15min":7.125},"max":{"1min":21.306000000000001,"5min":383.13999999999999,"15min":7477.2579999999998},"last":29.375}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":44.006999999999998,"5min":37.201999999999998,"15min":29.158999999999999},"min":{"1min":13.180999999999999,"5min":12.316000000000001,"15min":8.7569999999999997},"max":{"1min":386.452,"5min":386.452,"15min":386.452},"last":38.423000000000002},{"interface":"front","average":{"1min":31.984999999999999,"5min":28.108000000000001,"15min":25.620999999999999},"min":{"1min":8.2970000000000006,"5min":7.835,"15min":7.835},"max":{"1min":300.58199999999999,"5min":300.58199999999999,"15min":300.58199999999999},"last":37.646999999999998}]},{"osd":7,"last update":"Sat Mar 30 07:44:06 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":22.975999999999999,"15min":327.85500000000002},"min":{"1min":7.7320000000000002,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":18.007000000000001,"5min":295.38600000000002,"15min":21868.147000000001},"last":30.052},{"interface":"front","average":{"1min":13.711,"5min":24.344000000000001,"15min":331.459},"min":{"1min":10.34,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":18.896999999999998,"5min":302.11500000000001,"15min":21842.699000000001},"last":48.261000000000003}]}]},{"osd":4,"up_from":20,"seq":85899346251,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78628,"kb_used_data":7312,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293212,"statfs":{"total":96636764160,"available":96556249088,"internally_reserved":0,"allocated":7487488,"data_stored":4198149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":6.7249999999999996,"5min":24.440999999999999,"15min":22.800999999999998},"min":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":13.228,"5min":350.36599999999999,"15min":759.322},"last":172.70699999999999},{"interface":"front","average":{"1min":8.5169999999999995,"5min":27.239000000000001,"15min":28.228000000000002},"min":{"1min":5.9379999999999997,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":11.881,"5min":410.93099999999998,"15min":761.40899999999999},"last":53.140999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":10.500999999999999,"5min":21.305,"15min":21.925000000000001},"min":{"1min":4.6840000000000002,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":26.32,"5min":313.54399999999998,"15min":313.54399999999998},"last":65.491},{"interface":"front","average":{"1min":13.638999999999999,"5min":29.907,"15min":29.178000000000001},"min":{"1min":8.8420000000000005,"5min":8.8420000000000005,"15min":6.3719999999999999},"max":{"1min":27.036999999999999,"5min":466.54599999999999,"15min":466.54599999999999},"last":66.787000000000006}]},{"osd":2,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":9.2159999999999993,"5min":21.456,"15min":26.949999999999999},"min":{"1min":6.3659999999999997,"5min":6.1139999999999999,"15min":5.7800000000000002},"max":{"1min":11.869,"5min":371.74200000000002,"15min":768.79100000000005},"last":33.648000000000003},{"interface":"front","average":{"1min":13.162000000000001,"5min":25.972999999999999,"15min":23.812999999999999},"min":{"1min":9.3089999999999993,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":17.018000000000001,"5min":372.322,"15min":372.322},"last":37.347000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":12.989000000000001,"5min":413.31400000000002,"15min":188.03399999999999},"min":{"1min":8.3800000000000008,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":17.893999999999998,"5min":11317.004000000001,"15min":11317.004000000001},"last":46.061999999999998},{"interface":"front","average":{"1min":15.349,"5min":405.84899999999999,"15min":187.77000000000001},"min":{"1min":8.4019999999999992,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":46.728999999999999,"5min":11222.611999999999,"15min":11222.611999999999},"last":173.566}]},{"osd":5,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":13.702,"5min":27.013000000000002,"15min":43.438000000000002},"min":{"1min":10.589,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":16.962,"5min":421.40600000000001,"15min":4807.1080000000002},"last":88.423000000000002},{"interface":"front","average":{"1min":14.109999999999999,"5min":28.047000000000001,"15min":47.658000000000001},"min":{"1min":8.6690000000000005,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":17.809999999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":73.852000000000004}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":35.414000000000001,"5min":31.486000000000001,"15min":30.466999999999999},"min":{"1min":9.8079999999999998,"5min":9.4640000000000004,"15min":9.4640000000000004},"max":{"1min":223.85499999999999,"5min":529.13699999999994,"15min":529.13699999999994},"last":72.775999999999996},{"interface":"front","average":{"1min":38.140999999999998,"5min":34.231999999999999,"15min":36.503999999999998},"min":{"1min":13.925000000000001,"5min":11.499000000000001,"15min":7.1870000000000003},"max":{"1min":223.16999999999999,"5min":529.73000000000002,"15min":967.58399999999995},"last":118.96299999999999}]},{"osd":7,"last update":"Sat Mar 30 07:43:45 2024","interfaces":[{"interface":"back","average":{"1min":14.84,"5min":32.366999999999997,"15min":341.73399999999998},"min":{"1min":10.010999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":19.140999999999998,"5min":530.32500000000005,"15min":20845.164000000001},"last":30.667000000000002},{"interface":"front","average":{"1min":16.094999999999999,"5min":41.709000000000003,"15min":346.267},"min":{"1min":9.3520000000000003,"5min":8.1620000000000008,"15min":7.5060000000000002},"max":{"1min":19.088000000000001,"5min":664.02300000000002,"15min":20888.814999999999},"last":29.001000000000001}]}]},{"osd":7,"up_from":19,"seq":81604378951,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60604,"kb_used_data":7336,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311236,"statfs":{"total":96636764160,"available":96574705664,"internally_reserved":0,"allocated":7512064,"data_stored":4200908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":21.870999999999999,"15min":22.189},"min":{"1min":5.0609999999999999,"5min":4.8719999999999999,"15min":4.2789999999999999},"max":{"1min":51.43,"5min":358.94499999999999,"15min":368.16899999999998},"last":12.932},{"interface":"front","average":{"1min":11.212999999999999,"5min":25.902000000000001,"15min":24.701000000000001},"min":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":50.631,"5min":349.97199999999998,"15min":349.97199999999998},"last":6.8639999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":9.4149999999999991,"5min":14.821,"15min":24.390999999999998},"min":{"1min":4.577,"5min":4.577,"15min":4.577},"max":{"1min":16.036999999999999,"5min":94.453999999999994,"15min":472.33300000000003},"last":7.5700000000000003},{"interface":"front","average":{"1min":11.449,"5min":21.518000000000001,"15min":26.239000000000001},"min":{"1min":7.4370000000000003,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":14.566000000000001,"5min":184.33500000000001,"15min":292.92099999999999},"last":12.113}]},{"osd":2,"last update":"Sat Mar 30 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":15.978999999999999,"5min":27.795000000000002,"15min":31.538},"min":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":6.6449999999999996},"max":{"1min":77.424000000000007,"5min":341.96100000000001,"15min":400.43099999999998},"last":13.747},{"interface":"front","average":{"1min":17.446000000000002,"5min":33.427,"15min":34.75},"min":{"1min":10.714,"5min":10.554,"15min":7.5049999999999999},"max":{"1min":53.476999999999997,"5min":389.37200000000001,"15min":439.31999999999999},"last":15.32}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":896.39800000000002,"5min":450.06999999999999,"15min":204.417},"min":{"1min":11.375999999999999,"5min":11.375999999999999,"15min":7.5940000000000003},"max":{"1min":13240.941000000001,"5min":13240.941000000001,"15min":13240.941000000001},"last":14.526},{"interface":"front","average":{"1min":915.66300000000001,"5min":450.815,"15min":204.88300000000001},"min":{"1min":7.484,"5min":7.423,"15min":7.423},"max":{"1min":13596.368,"5min":13596.368,"15min":13596.368},"last":9.5570000000000004}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":11.497,"5min":349.65699999999998,"15min":225.608},"min":{"1min":7.0609999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":19.181999999999999,"5min":11792.466,"15min":13877.913},"last":16.128},{"interface":"front","average":{"1min":13.494999999999999,"5min":360.642,"15min":230.58600000000001},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":19.454000000000001,"5min":11966.236999999999,"15min":14034.805},"last":8.9589999999999996}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":24.239000000000001,"15min":108.086},"min":{"1min":11.356999999999999,"5min":7.6619999999999999,"15min":7.4050000000000002},"max":{"1min":21.242000000000001,"5min":345.90499999999997,"15min":7694.1369999999997},"last":17.125},{"interface":"front","average":{"1min":12.236000000000001,"5min":22.170000000000002,"15min":112.55800000000001},"min":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":18.193000000000001,"5min":203.31299999999999,"15min":7725.4070000000002},"last":11.071}]},{"osd":6,"last update":"Sat Mar 30 07:42:11 2024","interfaces":[{"interface":"back","average":{"1min":27.544,"5min":35.518999999999998,"15min":31.760999999999999},"min":{"1min":9.8190000000000008,"5min":8.7620000000000005,"15min":8.7620000000000005},"max":{"1min":171.96100000000001,"5min":316.012,"15min":316.012},"last":18.978000000000002},{"interface":"front","average":{"1min":35.613999999999997,"5min":36.930999999999997,"15min":29.056999999999999},"min":{"1min":18.420999999999999,"5min":6.8700000000000001,"15min":6.3090000000000002},"max":{"1min":194.72399999999999,"5min":743.27700000000004,"15min":743.27700000000004},"last":18.029}]}]},{"osd":1,"up_from":16,"seq":68719477077,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61016,"kb_used_data":7672,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310824,"statfs":{"total":96636764160,"available":96574283776,"internally_reserved":0,"allocated":7856128,"data_stored":4537313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":83,"apply_latency_ms":83,"commit_latency_ns":83000000,"apply_latency_ns":83000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":10.23,"5min":13.962,"15min":20.478000000000002},"min":{"1min":5.5880000000000001,"5min":5.2140000000000004,"15min":5.077},"max":{"1min":16.25,"5min":132.02699999999999,"15min":421.78699999999998},"last":6.0880000000000001},{"interface":"front","average":{"1min":9.8460000000000001,"5min":13.878,"15min":20.594999999999999},"min":{"1min":5.3019999999999996,"5min":5.2759999999999998,"15min":5.1139999999999999},"max":{"1min":13.212999999999999,"5min":116.801,"15min":409.91000000000003},"last":8.5020000000000007}]},{"osd":2,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":14.778,"15min":21.677},"min":{"1min":5.7489999999999997,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":18.859999999999999,"5min":180.42599999999999,"15min":433.34899999999999},"last":7.04},{"interface":"front","average":{"1min":11.426,"5min":18.071999999999999,"15min":23.861000000000001},"min":{"1min":8.8480000000000008,"5min":7.0700000000000003,"15min":5.6619999999999999},"max":{"1min":15.522,"5min":177.982,"15min":624.27300000000002},"last":15.009}]},{"osd":3,"last update":"Sat Mar 30 07:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1671.011,"5min":557.96100000000001,"15min":225.822},"min":{"1min":9.8309999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":12913.004000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":16.109999999999999},{"interface":"front","average":{"1min":1712.2249999999999,"5min":566.452,"15min":227.315},"min":{"1min":9.1899999999999995,"5min":7.1689999999999996,"15min":6.6130000000000004},"max":{"1min":13136.255999999999,"5min":13136.255999999999,"15min":13136.255999999999},"last":9.0869999999999997}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":18.109000000000002},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":9.6419999999999995}]},{"osd":5,"last update":"Sat Mar 30 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":17.719000000000001,"5min":20.428999999999998,"15min":47.131},"min":{"1min":12.308999999999999,"5min":9.8079999999999998,"15min":8.3439999999999994},"max":{"1min":44.511000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":17.315999999999999},{"interface":"front","average":{"1min":16.501999999999999,"5min":19.215,"15min":44.482999999999997},"min":{"1min":11.026,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":46.036000000000001,"5min":140.625,"15min":7345.2740000000003},"last":10.175000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:41:46 2024","interfaces":[{"interface":"back","average":{"1min":17.646999999999998,"5min":19.326000000000001,"15min":26.009},"min":{"1min":11.462,"5min":9.2889999999999997,"15min":7.8540000000000001},"max":{"1min":24.370000000000001,"5min":137.40199999999999,"15min":318.84100000000001},"last":11.667999999999999},{"interface":"front","average":{"1min":12.010999999999999,"5min":13.401999999999999,"15min":24.765000000000001},"min":{"1min":8.1199999999999992,"5min":7.7110000000000003,"15min":7.4210000000000003},"max":{"1min":18.332000000000001,"5min":71.405000000000001,"15min":452.00700000000001},"last":7.774}]},{"osd":7,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":14.731999999999999,"5min":16.477,"15min":243.119},"min":{"1min":9.6280000000000001,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":20.175999999999998,"5min":115.461,"15min":20279.343000000001},"last":18.882000000000001},{"interface":"front","average":{"1min":12.869999999999999,"5min":22.413,"15min":251.726},"min":{"1min":6.4550000000000001,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":17.719999999999999,"5min":281.423,"15min":20311.815999999999},"last":10.976000000000001}]}]},{"osd":5,"up_from":231,"seq":992137445697,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60956,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310884,"statfs":{"total":96636764160,"available":96574345216,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":10.680999999999999},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":6.3929999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":21.510000000000002},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":11.289}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":22.34},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":15.675000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":7.6020000000000003},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":23.135999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":11.842000000000001},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":17.524000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:38 2024","interfaces":[{"interface":"back","average":{"1min":30.094999999999999,"5min":22.585000000000001,"15min":27.227},"min":{"1min":14.298999999999999,"5min":11.044,"15min":8.9009999999999998},"max":{"1min":102.39400000000001,"5min":102.39400000000001,"15min":236.04599999999999},"last":16.920000000000002},{"interface":"front","average":{"1min":21.274000000000001,"5min":17.177,"15min":24.478999999999999},"min":{"1min":10.237,"5min":9.1620000000000008,"15min":6.4249999999999998},"max":{"1min":85.435000000000002,"5min":85.435000000000002,"15min":248.31299999999999},"last":25.350000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:13 2024","interfaces":[{"interface":"back","average":{"1min":11.587,"5min":15.007,"15min":283.82900000000001},"min":{"1min":7.585,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":18.324999999999999,"5min":115.589,"15min":19723.973000000002},"last":20.396000000000001},{"interface":"front","average":{"1min":15.435,"5min":19.620999999999999,"15min":283.815},"min":{"1min":10.228999999999999,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":19.495000000000001,"5min":95.783000000000001,"15min":19724.860000000001},"last":26.346}]}]},{"osd":3,"up_from":14,"seq":60129542477,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82692,"kb_used_data":7280,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289148,"statfs":{"total":96636764160,"available":96552087552,"internally_reserved":0,"allocated":7454720,"data_stored":4161164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":38.729999999999997},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":226.017}]},{"osd":1,"last update":"Sat Mar 30 07:44:28 2024","interfaces":[{"interface":"back","average":{"1min":34.786999999999999,"5min":22.965,"15min":21.632999999999999},"min":{"1min":8.6310000000000002,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":291.73599999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":227.48699999999999},{"interface":"front","average":{"1min":48.518999999999998,"5min":29.795999999999999,"15min":25.812000000000001},"min":{"1min":7.7089999999999996,"5min":7.5599999999999996,"15min":6.415},"max":{"1min":541.21199999999999,"5min":541.21199999999999,"15min":541.21199999999999},"last":159.37200000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":148.59399999999999},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":225.03899999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":158.285},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":248.155}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":231.13},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":278.62700000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:36 2024","interfaces":[{"interface":"back","average":{"1min":21.367999999999999,"5min":31.001999999999999,"15min":27.120000000000001},"min":{"1min":11.406000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":110.607,"5min":435.96600000000001,"15min":435.96600000000001},"last":172.33199999999999},{"interface":"front","average":{"1min":21.904,"5min":24.468,"15min":25.545999999999999},"min":{"1min":10.824,"5min":10.608000000000001,"15min":8.1180000000000003},"max":{"1min":62.828000000000003,"5min":145.239,"15min":259.60399999999998},"last":248.833}]},{"osd":7,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":39.734000000000002,"5min":26.948,"15min":285.68700000000001},"min":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":8.2690000000000001},"max":{"1min":527.00300000000004,"5min":527.00300000000004,"15min":20495.457999999999},"last":295.31599999999997},{"interface":"front","average":{"1min":30.413,"5min":28.954000000000001,"15min":287.50799999999998},"min":{"1min":11.025,"5min":11.025,"15min":11.025},"max":{"1min":308.75900000000001,"5min":308.75900000000001,"15min":20496.241000000002},"last":262.07799999999997}]}]},{"osd":6,"up_from":978,"seq":4200478015811,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7292,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7467008,"data_stored":4146127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":463.66800000000001,"5min":104.768,"15min":47.588000000000001},"min":{"1min":463.66800000000001,"5min":5.8390000000000004,"15min":4.6150000000000002},"max":{"1min":463.66800000000001,"5min":463.66800000000001,"15min":463.66800000000001},"last":11.76},{"interface":"front","average":{"1min":466.404,"5min":108.209,"15min":52.970999999999997},"min":{"1min":466.404,"5min":8.5449999999999999,"15min":5.8150000000000004},"max":{"1min":466.404,"5min":466.404,"15min":466.404},"last":7.5309999999999997}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":45.661999999999999,"5min":19.164999999999999,"15min":17.347000000000001},"min":{"1min":6.5149999999999997,"5min":5.4669999999999996,"15min":4.8579999999999997},"max":{"1min":298.72500000000002,"5min":298.72500000000002,"15min":298.72500000000002},"last":14.444000000000001},{"interface":"front","average":{"1min":53.417000000000002,"5min":25.199999999999999,"15min":22.443000000000001},"min":{"1min":8.4190000000000005,"5min":7.944,"15min":7.5519999999999996},"max":{"1min":323.78800000000001,"5min":323.78800000000001,"15min":323.78800000000001},"last":12.382}]},{"osd":2,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":314.89600000000002,"5min":76.939999999999998,"15min":39.972000000000001},"min":{"1min":314.89600000000002,"5min":6.351,"15min":6.351},"max":{"1min":314.89600000000002,"5min":314.89600000000002,"15min":314.89600000000002},"last":29.376999999999999},{"interface":"front","average":{"1min":353.24000000000001,"5min":87.257999999999996,"15min":46.240000000000002},"min":{"1min":353.24000000000001,"5min":8.266,"15min":8.266},"max":{"1min":353.24000000000001,"5min":353.24000000000001,"15min":353.24000000000001},"last":36.253}]},{"osd":3,"last update":"Sat Mar 30 07:42:29 2024","interfaces":[{"interface":"back","average":{"1min":22.446999999999999,"5min":188.88,"15min":130.78899999999999},"min":{"1min":9.8559999999999999,"5min":6.7830000000000004,"15min":6.7830000000000004},"max":{"1min":144.38300000000001,"5min":9266.1270000000004,"15min":9266.1270000000004},"last":8.4589999999999996},{"interface":"front","average":{"1min":16.998000000000001,"5min":177.846,"15min":123.959},"min":{"1min":7.3179999999999996,"5min":6.1200000000000001,"15min":4.8339999999999996},"max":{"1min":86.635000000000005,"5min":9082.3240000000005,"15min":9082.3240000000005},"last":18.457999999999998}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":331.70499999999998,"5min":557.46400000000006,"15min":326.00999999999999},"min":{"1min":331.70499999999998,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":331.70499999999998,"5min":13752.589,"15min":13752.589},"last":16.968},{"interface":"front","average":{"1min":469.358,"5min":575.79200000000003,"15min":334.21499999999997},"min":{"1min":469.358,"5min":7.5999999999999996,"15min":7.0199999999999996},"max":{"1min":469.358,"5min":13613.244000000001,"15min":13613.244000000001},"last":13.032999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":42.067999999999998,"5min":19.300999999999998,"15min":110.551},"min":{"1min":8.1010000000000009,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":483.904,"5min":483.904,"15min":9617.3179999999993},"last":16.385999999999999},{"interface":"front","average":{"1min":46.238,"5min":24.486000000000001,"15min":110.789},"min":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":8.8580000000000005},"max":{"1min":485.661,"5min":485.661,"15min":9386.0400000000009},"last":9.8219999999999992}]},{"osd":7,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":67.293000000000006,"5min":34.445,"15min":29.379999999999999},"min":{"1min":12.813000000000001,"5min":7.9729999999999999,"15min":7.9729999999999999},"max":{"1min":470.34300000000002,"5min":470.34300000000002,"15min":470.34300000000002},"last":17.635000000000002},{"interface":"front","average":{"1min":70.335999999999999,"5min":35.420000000000002,"15min":29.364000000000001},"min":{"1min":12.48,"5min":8.452,"15min":7.4189999999999996},"max":{"1min":487.30700000000002,"5min":487.30700000000002,"15min":487.30700000000002},"last":15.579000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:44:51.684 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.783+0000 7feef2bd0640 10 monclient: get_monmap_and_config 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.783+0000 7feef2bd0640 10 monclient: build_initial_monmap 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.783+0000 7feef2bd0640 10 monclient: monmap: 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:51.784042+0000 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:51.784042+0000 2024-03-30T07:44:51.785 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:51.786 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 Processor -- start 2024-03-30T07:44:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 -- start start 2024-03-30T07:44:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient: init 2024-03-30T07:44:51.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7feeec071be0 0x7feeec0a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-c con 0x7feeec071be0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 0x7feeec144a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-a con 0x7feeec0b0ca0 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feeec144f80 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeec0a8cc0 con 0x7feeec144f80 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeec071fc0 con 0x7feeec0b0ca0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeec0aa380 con 0x7feeec071be0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.784+0000 7feef2bd0640 10 monclient(hunting): authenticate will time out at 3217.899811s 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 0x7feeec144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feef096c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7feeec071be0 0x7feeec0a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 0x7feeec144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33082/0 (socket says 172.21.15.47:33082) 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 1 -- 172.21.15.47:0/1056764742 learned_addr learned my addr 172.21.15.47:0/1056764742 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 10 monclient(hunting): get_auth_request con 0x7feeec0b0ca0 auth_method 0 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feeebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feef096c640 10 monclient(hunting): get_auth_request con 0x7feeec071be0 auth_method 0 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feef096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feef096c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.785+0000 7feef096c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.786+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 10 monclient(hunting): handle_auth_done global_id 22324 payload 274 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 1 -- 172.21.15.47:0/1056764742 >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 msgr2=0x7feeec144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 1 --2- 172.21.15.47:0/1056764742 >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 0x7feeec144a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 1 -- 172.21.15.47:0/1056764742 >> v1:172.21.15.47:6789/0 conn(0x7feeec144f80 legacy=0x7feeec13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 10 monclient: _finish_hunting 0 2024-03-30T07:44:51.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 1 monclient: found mon.noname-c 2024-03-30T07:44:51.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:51.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.787+0000 7feef096c640 1 -- 172.21.15.47:0/1056764742 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeec0a8970 con 0x7feeec071be0 2024-03-30T07:44:51.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.789+0000 7feef096c640 1 --2- 172.21.15.47:0/1056764742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec071be0 0x7feeec0a9150 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 crypto rx=0x7feeec0b4b50 tx=0x7feee402be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fb61c99bb492fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 1 -- 172.21.15.47:0/1056764742 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee402cbf0 con 0x7feeec071be0 2024-03-30T07:44:51.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: dump: 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:51.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 1 -- 172.21.15.47:0/1056764742 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee402cd50 con 0x7feeec071be0 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feef2bd0640 5 monclient: authenticate success, global_id 22324 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 1 -- 172.21.15.47:0/1056764742 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4034660 con 0x7feeec071be0 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feeeb7fe640 10 monclient: dump: 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feef2bd0640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:51.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.793+0000 7feef2bd0640 4 set_mon_vals no callback set 2024-03-30T07:44:51.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 10 monclient: shutdown 2024-03-30T07:44:51.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 -- 172.21.15.47:0/1056764742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec071be0 msgr2=0x7feeec0a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1056764742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec071be0 0x7feeec0a9150 secure :-1 s=READY pgs=5508 cs=0 l=1 rev1=1 crypto rx=0x7feeec0b4b50 tx=0x7feee402be40 comp rx=0 tx=0).stop 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 -- 172.21.15.47:0/1056764742 shutdown_connections 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1056764742 >> v2:172.21.15.47:3300/0 conn(0x7feeec0b0ca0 0x7feeec144a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1056764742 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec071be0 0x7feeec0a9150 unknown :-1 s=CLOSED pgs=5508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 -- 172.21.15.47:0/1056764742 >> 172.21.15.47:0/1056764742 conn(0x7feeec0b73d0 msgr2=0x7feeec0b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 -- 172.21.15.47:0/1056764742 shutdown_connections 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.794+0000 7feef2bd0640 1 -- 172.21.15.47:0/1056764742 wait complete. 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 10 monclient: build_initial_monmap 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 10 monclient: monmap: 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:51.802 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 librados: starting msgr at 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 librados: starting objecter 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 Processor -- start 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 -- start start 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 librados: setting wanted keys 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 1 librados: calling monclient init 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.795+0000 7feef2bd0640 10 monclient: init 2024-03-30T07:44:51.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feeec071be0 0x7feeec133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-c con 0x7feeec071be0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feeec0b0ca0 0x7feeec0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-a con 0x7feeec0b0ca0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): picked mon.noname-b con 0x7feeec144f80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b6d30 con 0x7feeec0b0ca0 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b6960 con 0x7feeec144f80 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0b6ad0 con 0x7feeec071be0 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.796+0000 7feef2bd0640 10 monclient(hunting): authenticate will time out at 3217.911649s 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feeebfff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feeec0b0ca0 0x7feeec0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef096c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feeec071be0 0x7feeec133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38426/0 (socket says 172.21.15.47:38426) 2024-03-30T07:44:51.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 1 -- 172.21.15.47:0/1829764555 learned_addr learned my addr 172.21.15.47:0/1829764555 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): get_auth_request con 0x7feeec144f80 auth_method 0 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef096c640 10 monclient(hunting): get_auth_request con 0x7feeec071be0 auth_method 0 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef096c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef096c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef096c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feeebfff640 10 monclient(hunting): get_auth_request con 0x7feeec0b0ca0 auth_method 0 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feeebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feeebfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feeebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.797+0000 7feef116d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feeebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:51.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:51.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.798+0000 7feef096c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 10 monclient(hunting): handle_auth_done global_id 22327 payload 824 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 -- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feeec071be0 msgr2=0x7feeec133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feeec071be0 0x7feeec133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 -- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feeec0b0ca0 msgr2=0x7feeec0b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feeec0b0ca0 0x7feeec0b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 10 monclient: _finish_hunting 0 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 monclient: found mon.noname-b 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.800+0000 7feef116d640 1 -- 172.21.15.47:0/1829764555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4009a70 con 0x7feeec144f80 2024-03-30T07:44:51.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.802+0000 7feef116d640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 secure :-1 s=READY pgs=5509 cs=0 l=1 rev1=1 crypto rx=0x7feedc00c410 tx=0x7feedc00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.802+0000 7feee97fa640 10 client.?.objecter ms_handle_connect 0x7feeec144f80 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.802+0000 7feee97fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc011790 con 0x7feeec144f80 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: dump: 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.810 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: _finish_auth 0 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: _check_auth_tickets 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc0118f0 con 0x7feeec144f80 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc011ce0 con 0x7feeec144f80 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 5 monclient: authenticate success, global_id 22327 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 10 monclient: _renew_subs 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec136810 con 0x7feeec144f80 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feeeaffd640 4 set_mon_vals no callback set 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feee97fa640 10 monclient: dump: 2024-03-30T07:44:51.811 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 10 client.22327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 10 monclient: _renew_subs 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec0b4d80 con 0x7feeec144f80 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.806+0000 7feef2bd0640 1 librados: init done 2024-03-30T07:44:51.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.807+0000 7feef2bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:51.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.807+0000 7feef2bd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:51.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.807+0000 7feef2bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:51.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.807+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec137ce0 con 0x7feeec144f80 2024-03-30T07:44:51.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.827+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feedc01c460 con 0x7feeec144f80 2024-03-30T07:44:51.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.827+0000 7feee97fa640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 0x7feecc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:51.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.827+0000 7feef096c640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 0x7feecc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:51.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.827+0000 7feef096c640 10 monclient: get_auth_request con 0x7feecc06c5e0 auth_method 0 2024-03-30T07:44:51.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.828+0000 7feef096c640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 0x7feecc06eaa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feee4036000 tx=0x7feee4030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:51.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.828+0000 7feee97fa640 10 client.22327.objecter ms_handle_connect 0x7feecc06c5e0 2024-03-30T07:44:51.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.829+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 473..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7feedc011e40 con 0x7feeec144f80 2024-03-30T07:44:51.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.829+0000 7feee97fa640 10 client.22327.objecter ms_dispatch 0x7feeec000b90 osd_map(994..994 src has 473..994) v4 2024-03-30T07:44:51.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.829+0000 7feee97fa640 3 client.22327.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:51.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.829+0000 7feee97fa640 3 client.22327.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:51.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:51.829+0000 7feee97fa640 20 client.22327.objecter dump_active .. 0 homeless 2024-03-30T07:44:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.085+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7feedc054ce0 con 0x7feeec144f80 2024-03-30T07:44:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.085+0000 7feee97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:52.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.085+0000 7feee97fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:52.191 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:20.002 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.197 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:20.007 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.214 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:20.025 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.248 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:20.025 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.251 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:20.029 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.269 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:20.046 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.273 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:20.083 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.278 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:20.088 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.296 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:20.107 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:44:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.337+0000 7feef2bd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:44:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.337+0000 7feef2bd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:44:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.337+0000 7feef2bd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:52.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.337+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeec137ce0 con 0x7feeec144f80 2024-03-30T07:44:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.453+0000 7feee97fa640 1 -- 172.21.15.47:0/1829764555 <== mon.1 v2:172.21.15.52:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v994) v1 ==== 74+0+17135 (secure 0 0 0) 0x7feeec137ce0 con 0x7feeec144f80 2024-03-30T07:44:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.453+0000 7feee97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-30T07:44:52.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.453+0000 7feee97fa640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:44:52.456 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:44:52.456 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":994,"fsid":"bc5c1745-fe46-4846-b9b2-29d88e4530d9","created":"2024-03-30T07:15:44.075124+0000","modified":"2024-03-30T07:44:49.097554+0000","last_up_change":"2024-03-30T07:44:16.538133+0000","last_in_change":"2024-03-30T07:15:50.083237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":260,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T07:16:32.182687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T07:16:47.206094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-30T07:17:51.516962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bf658e45-24f6-490b-825c-7331119bd2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":35205},{"type":"v1","addr":"172.21.15.47:6801","nonce":35205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":35205},{"type":"v1","addr":"172.21.15.47:6803","nonce":35205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":35205},{"type":"v1","addr":"172.21.15.47:6807","nonce":35205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":35205},{"type":"v1","addr":"172.21.15.47:6805","nonce":35205}]},"public_addr":"172.21.15.47:6801/35205","cluster_addr":"172.21.15.47:6803/35205","heartbeat_back_addr":"172.21.15.47:6807/35205","heartbeat_front_addr":"172.21.15.47:6805/35205","state":["exists","up"]},{"osd":1,"uuid":"1a3ae202-ca24-4165-affc-cf9ee8066e8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":35204},{"type":"v1","addr":"172.21.15.47:6809","nonce":35204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":35204},{"type":"v1","addr":"172.21.15.47:6813","nonce":35204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":35204},{"type":"v1","addr":"172.21.15.47:6820","nonce":35204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":35204},{"type":"v1","addr":"172.21.15.47:6816","nonce":35204}]},"public_addr":"172.21.15.47:6809/35204","cluster_addr":"172.21.15.47:6813/35204","heartbeat_back_addr":"172.21.15.47:6820/35204","heartbeat_front_addr":"172.21.15.47:6816/35204","state":["exists","up"]},{"osd":2,"uuid":"b93375d8-53af-4822-ac8f-a660429e26f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":35207},{"type":"v1","addr":"172.21.15.47:6825","nonce":35207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":35207},{"type":"v1","addr":"172.21.15.47:6827","nonce":35207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":35207},{"type":"v1","addr":"172.21.15.47:6831","nonce":35207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":35207},{"type":"v1","addr":"172.21.15.47:6829","nonce":35207}]},"public_addr":"172.21.15.47:6825/35207","cluster_addr":"172.21.15.47:6827/35207","heartbeat_back_addr":"172.21.15.47:6831/35207","heartbeat_front_addr":"172.21.15.47:6829/35207","state":["exists","up"]},{"osd":3,"uuid":"8fc299ae-e44c-42d3-aa0c-6f7cafad40ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":35206},{"type":"v1","addr":"172.21.15.47:6811","nonce":35206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6815","nonce":35206},{"type":"v1","addr":"172.21.15.47:6817","nonce":35206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":35206},{"type":"v1","addr":"172.21.15.47:6823","nonce":35206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6819","nonce":35206},{"type":"v1","addr":"172.21.15.47:6821","nonce":35206}]},"public_addr":"172.21.15.47:6811/35206","cluster_addr":"172.21.15.47:6817/35206","heartbeat_back_addr":"172.21.15.47:6823/35206","heartbeat_front_addr":"172.21.15.47:6821/35206","state":["exists","up"]},{"osd":4,"uuid":"4e1deafb-c667-4984-8939-c657168e85da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":35313},{"type":"v1","addr":"172.21.15.52:6819","nonce":35313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":35313},{"type":"v1","addr":"172.21.15.52:6821","nonce":35313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":35313},{"type":"v1","addr":"172.21.15.52:6825","nonce":35313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":35313},{"type":"v1","addr":"172.21.15.52:6823","nonce":35313}]},"public_addr":"172.21.15.52:6819/35313","cluster_addr":"172.21.15.52:6821/35313","heartbeat_back_addr":"172.21.15.52:6825/35313","heartbeat_front_addr":"172.21.15.52:6823/35313","state":["exists","up"]},{"osd":5,"uuid":"1c5fc6e0-c7f9-41df-b24d-6150802b1e66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":228,"up_from":231,"up_thru":989,"down_at":217,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6826","nonce":35314},{"type":"v1","addr":"172.21.15.52:6827","nonce":35314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6834","nonce":1035314},{"type":"v1","addr":"172.21.15.52:6835","nonce":1035314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6832","nonce":35314},{"type":"v1","addr":"172.21.15.52:6833","nonce":35314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6830","nonce":35314},{"type":"v1","addr":"172.21.15.52:6831","nonce":35314}]},"public_addr":"172.21.15.52:6827/35314","cluster_addr":"172.21.15.52:6835/1035314","heartbeat_back_addr":"172.21.15.52:6833/35314","heartbeat_front_addr":"172.21.15.52:6831/35314","state":["exists","up"]},{"osd":6,"uuid":"4aa56a54-cd05-4f40-b7e3-a841be97681c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":977,"up_from":978,"up_thru":989,"down_at":971,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":35315},{"type":"v1","addr":"172.21.15.52:6803","nonce":35315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6828","nonce":1035315},{"type":"v1","addr":"172.21.15.52:6829","nonce":1035315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":35315},{"type":"v1","addr":"172.21.15.52:6813","nonce":35315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":35315},{"type":"v1","addr":"172.21.15.52:6809","nonce":35315}]},"public_addr":"172.21.15.52:6803/35315","cluster_addr":"172.21.15.52:6829/1035315","heartbeat_back_addr":"172.21.15.52:6813/35315","heartbeat_front_addr":"172.21.15.52:6809/35315","state":["exists","up"]},{"osd":7,"uuid":"09d844bd-e98e-4d61-a0ad-3c711bfe0c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":35316},{"type":"v1","addr":"172.21.15.52:6806","nonce":35316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":35316},{"type":"v1","addr":"172.21.15.52:6812","nonce":35316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":35316},{"type":"v1","addr":"172.21.15.52:6817","nonce":35316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":35316},{"type":"v1","addr":"172.21.15.52:6815","nonce":35316}]},"public_addr":"172.21.15.52:6806/35316","cluster_addr":"172.21.15.52:6812/35316","heartbeat_back_addr":"172.21.15.52:6817/35316","heartbeat_front_addr":"172.21.15.52:6815/35316","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.613520+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.651813+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.749978+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:29.080129+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.393136+0000","dead_epoch":0},{"osd":5,"down_stamp":"2024-03-30T07:23:57.047147+0000","laggy_probability":0.30000001192092896,"laggy_interval":4,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.240245+0000","dead_epoch":227},{"osd":6,"down_stamp":"2024-03-30T07:44:04.295245+0000","laggy_probability":0.30000001192092896,"laggy_interval":3,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:28.041599+0000","dead_epoch":976},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T07:16:27.954523+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.a","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":10,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":13,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":56,"snaps":[{"begin":11,"length":2}]},{"pool":84,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":94,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":106,"snaps":[{"begin":2,"length":1}]},{"pool":110,"snaps":[{"begin":7,"length":1}]},{"pool":117,"snaps":[{"begin":4,"length":2}]},{"pool":122,"snaps":[{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":178,"snaps":[{"begin":12,"length":1}]},{"pool":187,"snaps":[{"begin":14,"length":1}]},{"pool":192,"snaps":[{"begin":17,"length":1}]},{"pool":195,"snaps":[{"begin":19,"length":2}]},{"pool":198,"snaps":[{"begin":24,"length":1}]},{"pool":241,"snaps":[{"begin":2,"length":1}]},{"pool":242,"snaps":[{"begin":4,"length":1}]},{"pool":245,"snaps":[{"begin":6,"length":1}]},{"pool":249,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 10 librados: watch_flush enter 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 10 librados: watch_flush exit 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 20 client.22327.objecter shutdown clearing up homeless session... 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 10 client.22327.objecter successfully canceled tick 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 msgr2=0x7feecc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 0x7feecc06eaa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feee4036000 tx=0x7feee4030000 comp rx=0 tx=0).stop 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 10 monclient: shutdown 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 msgr2=0x7feeec1382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 secure :-1 s=READY pgs=5509 cs=0 l=1 rev1=1 crypto rx=0x7feedc00c410 tx=0x7feedc00c810 comp rx=0 tx=0).stop 2024-03-30T07:44:52.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 shutdown_connections 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7feecc06c5e0 0x7feecc06eaa0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7feeec144f80 0x7feeec1382a0 unknown :-1 s=CLOSED pgs=5509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feeec0b0ca0 0x7feeec0b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 --2- 172.21.15.47:0/1829764555 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feeec071be0 0x7feeec133de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 >> 172.21.15.47:0/1829764555 conn(0x7feeec0b73d0 msgr2=0x7feeec068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 shutdown_connections 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 -- 172.21.15.47:0/1829764555 wait complete. 2024-03-30T07:44:52.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:52.456+0000 7feef2bd0640 1 librados: shutdown 2024-03-30T07:44:53.475 INFO:tasks.ceph:Scrubbing osd.0 2024-03-30T07:44:53.475 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.578+0000 7f6372b5c640 10 monclient: get_monmap_and_config 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.578+0000 7f6372b5c640 10 monclient: build_initial_monmap 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.578+0000 7f6372b5c640 10 monclient: monmap: 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:53.579103+0000 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:53.579103+0000 2024-03-30T07:44:53.580 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:53.581 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 Processor -- start 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 -- start start 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient: init 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-30T07:44:53.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f636c073680 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f636c13e7c0 0x7f636c13ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-c con 0x7f636c13e7c0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f636c1449c0 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0aedc0 con 0x7f636c073680 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c0aeaa0 con 0x7f636c1449c0 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636c0712e0 con 0x7f636c13e7c0 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.579+0000 7f6372b5c640 10 monclient(hunting): authenticate will time out at 3219.694855s 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.580+0000 7f63708f8640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f636c073680 0x7f636c0ae390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55232/0 (socket says 172.21.15.47:55232) 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.580+0000 7f63708f8640 1 -- 172.21.15.47:0/3407411406 learned_addr learned my addr 172.21.15.47:0/3407411406 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:53.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.580+0000 7f636bfff640 1 --2- 172.21.15.47:0/3407411406 >> v2:172.21.15.52:3300/0 conn(0x7f636c13e7c0 0x7f636c13ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.582+0000 7f636bfff640 10 monclient(hunting): get_auth_request con 0x7f636c13e7c0 auth_method 0 2024-03-30T07:44:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.582+0000 7f636bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.582+0000 7f636bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.582+0000 7f636bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438721292 0 0) 0x7f636c0aedc0 con 0x7f636c073680 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 10 monclient(hunting): _negotiate 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 10 monclient(hunting): my global_id is 15834 2024-03-30T07:44:53.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.583+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f635c003520 con 0x7f636c073680 2024-03-30T07:44:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.584+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.584+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.584+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2636385867 0 0) 0x7f635c003520 con 0x7f636c073680 2024-03-30T07:44:53.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 >> v2:172.21.15.52:3300/0 conn(0x7f636c13e7c0 msgr2=0x7f636c13ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 --2- 172.21.15.47:0/3407411406 >> v2:172.21.15.52:3300/0 conn(0x7f636c13e7c0 0x7f636c13ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 >> v1:172.21.15.52:6789/0 conn(0x7f636c1449c0 legacy=0x7f636c144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 monclient: found mon.noname-a-legacy 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.47:6789/0 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0aff60 con 0x7f636c073680 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f6360001670 con 0x7f636c073680 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f6372b5c640 5 monclient: authenticate success, global_id 15834 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f6372b5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:53.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: dump: 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] but i'm connected to v1:172.21.15.47:6789/0, reconnecting 2024-03-30T07:44:53.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 >> v1:172.21.15.47:6789/0 conn(0x7f636c073680 legacy=0x7f636c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f635c0039c0 0x7f635c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): picked mon.b con 0x7f635c0039c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f635c0047c0 0x7f635c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): picked mon.a con 0x7f635c0047c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f635c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): picked mon.c con 0x7f635c005180 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c0712e0 con 0x7f635c0047c0 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c00c080 con 0x7f635c0039c0 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c00c240 con 0x7f635c005180 2024-03-30T07:44:53.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f636b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.586+0000 7f6372b5c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f635c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): get_auth_request con 0x7f635c005180 auth_method 0 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.587+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.589+0000 7f636bfff640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f635c0047c0 0x7f635c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.589+0000 7f63710f9640 10 monclient(hunting): handle_auth_done global_id 15834 payload 293 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 -- 172.21.15.47:0/3407411406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f635c0039c0 msgr2=0x7f635c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f635c0039c0 0x7f635c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 -- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f635c0047c0 msgr2=0x7f635c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:53.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f635c0047c0 0x7f635c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 10 monclient: _finish_hunting 0 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 monclient: found mon.c 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 -- 172.21.15.47:0/3407411406 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c003520 con 0x7f635c005180 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 10 monclient: _finish_auth 0 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 10 monclient: _check_auth_tickets 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.590+0000 7f63710f9640 1 -- 172.21.15.47:0/3407411406 reap_dead start 2024-03-30T07:44:53.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.592+0000 7f63710f9640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f635c00b980 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f6364001e50 tx=0x7f635c0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31b8d439f43c90b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364003b70 con 0x7f635c005180 2024-03-30T07:44:53.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 10 monclient: dump: 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364003cf0 con 0x7f635c005180 2024-03-30T07:44:53.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.595+0000 7f636b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f636b7fe640 1 -- 172.21.15.47:0/3407411406 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364005310 con 0x7f635c005180 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 4 set_mon_vals no callback set 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f636b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f636b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f636b7fe640 10 monclient: dump: 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.599 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.600 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 10 monclient: shutdown 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 -- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 msgr2=0x7f635c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f635c00b980 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f6364001e50 tx=0x7f635c0057f0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 -- 172.21.15.47:0/3407411406 shutdown_connections 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 --2- 172.21.15.47:0/3407411406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f635c00b980 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 -- 172.21.15.47:0/3407411406 >> 172.21.15.47:0/3407411406 conn(0x7f636c0b3f00 msgr2=0x7f636c069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 -- 172.21.15.47:0/3407411406 shutdown_connections 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.596+0000 7f6372b5c640 1 -- 172.21.15.47:0/3407411406 wait complete. 2024-03-30T07:44:53.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.597+0000 7f6372b5c640 10 monclient: build_initial_monmap 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.597+0000 7f6372b5c640 10 monclient: monmap: 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.597+0000 7f6372b5c640 1 librados: starting msgr at 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.597+0000 7f6372b5c640 1 librados: starting objecter 2024-03-30T07:44:53.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.597+0000 7f6372b5c640 1 Processor -- start 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 -- start start 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 librados: setting wanted keys 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 librados: calling monclient init 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient: init 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f636c1d9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-c con 0x7f635c005180 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636c1cca10 0x7f636c141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-a con 0x7f636c1cca10 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f636c141e40 0x7f636c13f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): picked mon.noname-b con 0x7f636c141e40 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1da190 con 0x7f636c1cca10 2024-03-30T07:44:53.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1414e0 con 0x7f636c141e40 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c1cbe40 con 0x7f635c005180 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.598+0000 7f6372b5c640 10 monclient(hunting): authenticate will time out at 3219.713913s 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f636c1d9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f636bfff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636c1cca10 0x7f636c141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63710f9640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f636c141e40 0x7f636c13f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f636c1d9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:59992/0 (socket says 172.21.15.47:59992) 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 1 -- 172.21.15.47:0/412265373 learned_addr learned my addr 172.21.15.47:0/412265373 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 10 monclient(hunting): get_auth_request con 0x7f635c005180 auth_method 0 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63708f8640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f636bfff640 10 monclient(hunting): get_auth_request con 0x7f636c1cca10 auth_method 0 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f636bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f636bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f636bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63710f9640 10 monclient(hunting): get_auth_request con 0x7f636c141e40 auth_method 0 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63710f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63710f9640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.599+0000 7f63710f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63708f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63708f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63708f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f63710f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.600+0000 7f636bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.602+0000 7f63710f9640 10 monclient(hunting): handle_auth_done global_id 22336 payload 824 2024-03-30T07:44:53.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 -- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 msgr2=0x7f636c1d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f636c1d9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 -- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636c1cca10 msgr2=0x7f636c141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636c1cca10 0x7f636c141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 10 monclient: _finish_hunting 0 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 monclient: found mon.noname-b 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63710f9640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c1422c0 con 0x7f636c141e40 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.603+0000 7f63708f8640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f635c005180 0x7f636c1d9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.605+0000 7f63710f9640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f636c141e40 0x7f636c13f240 secure :-1 s=READY pgs=5511 cs=0 l=1 rev1=1 crypto rx=0x7f636c0b1340 tx=0x7f63640014a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.605+0000 7f63697fa640 10 client.?.objecter ms_handle_connect 0x7f636c141e40 2024-03-30T07:44:53.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.605+0000 7f63697fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.608+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364006d50 con 0x7f636c141e40 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: dump: 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.612 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: _finish_auth 0 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: _check_auth_tickets 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 5 monclient: authenticate success, global_id 22336 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364007030 con 0x7f636c141e40 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63640073e0 con 0x7f636c141e40 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f636affd640 4 set_mon_vals no callback set 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f63697fa640 10 monclient: dump: 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.613 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 monclient: _renew_subs 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c140380 con 0x7f636c141e40 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 client.22336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 monclient: _renew_subs 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c13f780 con 0x7f636c141e40 2024-03-30T07:44:53.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 1 librados: init done 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 client.22336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 15 client.22336.objecter _session_command_op_assign -1 1 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 20 client.22336.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 client.22336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.609+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f636c141650 con 0x7f636c141e40 2024-03-30T07:44:53.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.630+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f636403f070 con 0x7f636c141e40 2024-03-30T07:44:53.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.630+0000 7f63697fa640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f635006c610 0x7f635006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.630+0000 7f63708f8640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f635006c610 0x7f635006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.630+0000 7f63708f8640 10 monclient: get_auth_request con 0x7f635006c610 auth_method 0 2024-03-30T07:44:53.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.631+0000 7f63708f8640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f635006c610 0x7f635006ead0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6360007e80 tx=0x7f6360001930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.631+0000 7f63697fa640 10 client.22336.objecter ms_handle_connect 0x7f635006c610 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f636407a1e0 con 0x7f636c141e40 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 10 client.22336.objecter ms_dispatch 0x7f636c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 3 client.22336.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 3 client.22336.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 10 client.22336.objecter checking command 1 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6350073fe0 0x7f6350076480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter _get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 15 client.22336.objecter _session_command_op_remove -1 1 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter _get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 15 client.22336.objecter _session_command_op_assign 0 1 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 4 2024-03-30T07:44:53.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 10 client.22336.objecter _send_command 1 2024-03-30T07:44:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6350076b50 con 0x7f6350073fe0 2024-03-30T07:44:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f63697fa640 20 client.22336.objecter dump_active .. 0 homeless 2024-03-30T07:44:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f636bfff640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6350073fe0 0x7f6350076480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.633+0000 7f636bfff640 10 monclient: get_auth_request con 0x7f6350073fe0 auth_method 0 2024-03-30T07:44:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.636+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6364076520 con 0x7f636c141e40 2024-03-30T07:44:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.636+0000 7f63697fa640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.636+0000 7f636bfff640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6350073fe0 0x7f6350076480 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.636+0000 7f63697fa640 10 client.22336.objecter ms_handle_connect 0x7f6350073fe0 2024-03-30T07:44:53.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.663+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== osd.0 v2:172.21.15.47:6800/35205 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6350076b50 con 0x7f6350073fe0 2024-03-30T07:44:53.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.663+0000 7f63697fa640 10 client.22336.objecter ms_dispatch 0x7f636c000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:53.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.663+0000 7f63697fa640 10 client.22336.objecter _finish_command 1 = system:0 2024-03-30T07:44:53.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.663+0000 7f63697fa640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 4 2024-03-30T07:44:53.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.663+0000 7f63697fa640 15 client.22336.objecter _session_command_op_remove 0 1 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 10 client.22336.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 15 client.22336.objecter _session_command_op_assign -1 2 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 20 client.22336.objecter _get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 20 client.22336.objecter _get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 15 client.22336.objecter _session_command_op_remove -1 2 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 20 client.22336.objecter get_session s=0x7f6350073910 osd=0 3 2024-03-30T07:44:53.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 15 client.22336.objecter _session_command_op_assign 0 2 2024-03-30T07:44:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 4 2024-03-30T07:44:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 10 client.22336.objecter _send_command 2 2024-03-30T07:44:53.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.676+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 --> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f636c1da800 con 0x7f6350073fe0 2024-03-30T07:44:53.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.692+0000 7f63697fa640 1 -- 172.21.15.47:0/412265373 <== osd.0 v2:172.21.15.47:6800/35205 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6358009810 con 0x7f6350073fe0 2024-03-30T07:44:53.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.692+0000 7f63697fa640 10 client.22336.objecter ms_dispatch 0x7f636c000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:53.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.692+0000 7f63697fa640 10 client.22336.objecter _finish_command 2 = system:0 2024-03-30T07:44:53.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.692+0000 7f63697fa640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 4 2024-03-30T07:44:53.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.692+0000 7f63697fa640 15 client.22336.objecter _session_command_op_remove 0 2 2024-03-30T07:44:53.695 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:53.695 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:53.695 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:53.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 10 librados: watch_flush enter 2024-03-30T07:44:53.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 10 librados: watch_flush exit 2024-03-30T07:44:53.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 10 client.22336.objecter close_session for osd.0 2024-03-30T07:44:53.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6350073fe0 msgr2=0x7f6350076480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.47:6800/35205,v1:172.21.15.47:6801/35205] conn(0x7f6350073fe0 0x7f6350076480 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 20 client.22336.objecter put_session s=0x7f6350073910 osd=0 1 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 20 client.22336.objecter shutdown clearing up homeless session... 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 10 client.22336.objecter successfully canceled tick 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f635006c610 msgr2=0x7f635006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f635006c610 0x7f635006ead0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6360007e80 tx=0x7f6360001930 comp rx=0 tx=0).stop 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 10 monclient: shutdown 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f636c141e40 msgr2=0x7f636c13f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 --2- 172.21.15.47:0/412265373 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f636c141e40 0x7f636c13f240 secure :-1 s=READY pgs=5511 cs=0 l=1 rev1=1 crypto rx=0x7f636c0b1340 tx=0x7f63640014a0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f63710f9640 1 -- 172.21.15.47:0/412265373 reap_dead start 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 shutdown_connections 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 >> 172.21.15.47:0/412265373 conn(0x7f636c0b3f00 msgr2=0x7f636c13df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 shutdown_connections 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 -- 172.21.15.47:0/412265373 wait complete. 2024-03-30T07:44:53.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.693+0000 7f6372b5c640 1 librados: shutdown 2024-03-30T07:44:53.706 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-30T07:44:53.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.808+0000 7f73e71f0640 10 monclient: get_monmap_and_config 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.808+0000 7f73e71f0640 10 monclient: build_initial_monmap 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.808+0000 7f73e71f0640 10 monclient: monmap: 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:53.809657+0000 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:53.809657+0000 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:53.811 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:53.812 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:53.812 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:53.812 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.809+0000 7f73e71f0640 1 Processor -- start 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.809+0000 7f73e71f0640 1 -- start start 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.809+0000 7f73e71f0640 10 monclient: init 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73e0071be0 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f73e00b0ca0 0x7f73e00a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-a con 0x7f73e00b0ca0 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73e013e820 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00a8cc0 con 0x7f73e013e820 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e00aa380 con 0x7f73e0071be0 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e0071fc0 con 0x7f73e00b0ca0 2024-03-30T07:44:53.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e71f0640 10 monclient(hunting): authenticate will time out at 3219.925450s 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e4f8c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f73e0071be0 0x7f73e0144a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:33642/0 (socket says 172.21.15.47:33642) 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73e4f8c640 1 -- 172.21.15.47:0/2082946225 learned_addr learned my addr 172.21.15.47:0/2082946225 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.810+0000 7f73dffff640 1 --2- 172.21.15.47:0/2082946225 >> v2:172.21.15.47:3300/0 conn(0x7f73e00b0ca0 0x7f73e00a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.811+0000 7f73dffff640 10 monclient(hunting): get_auth_request con 0x7f73e00b0ca0 auth_method 0 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.811+0000 7f73dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.811+0000 7f73dffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.811+0000 7f73dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 1 -- 172.21.15.47:0/2082946225 <== mon.1 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543399412 0 0) 0x7f73e00aa380 con 0x7f73e0071be0 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 10 monclient(hunting): _negotiate 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 10 monclient(hunting): my global_id is 22339 2024-03-30T07:44:53.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.812+0000 7f73df7fe640 1 -- 172.21.15.47:0/2082946225 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc003520 con 0x7f73e0071be0 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 10 monclient(hunting): handle_auth_done global_id 15840 payload 274 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 1 -- 172.21.15.47:0/2082946225 >> v1:172.21.15.52:6789/0 conn(0x7f73e0071be0 legacy=0x7f73e0144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 1 -- 172.21.15.47:0/2082946225 >> v1:172.21.15.47:6789/0 conn(0x7f73e013e820 legacy=0x7f73e014bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 10 monclient: _finish_hunting 0 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 1 monclient: found mon.noname-a 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:53.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.814+0000 7f73dffff640 1 -- 172.21.15.47:0/2082946225 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00a8970 con 0x7f73e00b0ca0 2024-03-30T07:44:53.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.816+0000 7f73dffff640 1 --2- 172.21.15.47:0/2082946225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e00a9150 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009d90 tx=0x7f73d002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58241753f2925e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 1 -- 172.21.15.47:0/2082946225 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d002cd00 con 0x7f73e00b0ca0 2024-03-30T07:44:53.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:53.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: dump: 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 1 -- 172.21.15.47:0/2082946225 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d002ce60 con 0x7f73e00b0ca0 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73e71f0640 5 monclient: authenticate success, global_id 15840 2024-03-30T07:44:53.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73e71f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73e71f0640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73e71f0640 4 set_mon_vals no callback set 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 1 -- 172.21.15.47:0/2082946225 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0034660 con 0x7f73e00b0ca0 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.822+0000 7f73df7fe640 10 monclient: dump: 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.826 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.827 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.827 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.827 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.827 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 10 monclient: shutdown 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 -- 172.21.15.47:0/2082946225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 msgr2=0x7f73e00a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2082946225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e00a9150 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009d90 tx=0x7f73d002bf50 comp rx=0 tx=0).stop 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 -- 172.21.15.47:0/2082946225 shutdown_connections 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2082946225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e00a9150 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 -- 172.21.15.47:0/2082946225 >> 172.21.15.47:0/2082946225 conn(0x7f73e00b73d0 msgr2=0x7f73e00b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 -- 172.21.15.47:0/2082946225 shutdown_connections 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.823+0000 7f73e71f0640 1 -- 172.21.15.47:0/2082946225 wait complete. 2024-03-30T07:44:53.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 10 monclient: build_initial_monmap 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 10 monclient: monmap: 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 librados: starting msgr at 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 librados: starting objecter 2024-03-30T07:44:53.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 Processor -- start 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 -- start start 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 librados: setting wanted keys 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 1 librados: calling monclient init 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.824+0000 7f73e71f0640 10 monclient: init 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f73e0071be0 0x7f73e01cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-c con 0x7f73e0071be0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e0141520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-a con 0x7f73e00b0ca0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): picked mon.noname-b con 0x7f73e013e820 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e013f4b0 con 0x7f73e00b0ca0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0140e20 con 0x7f73e013e820 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0141100 con 0x7f73e0071be0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e71f0640 10 monclient(hunting): authenticate will time out at 3219.940374s 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e4f8c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f73e0071be0 0x7f73e01cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73e578d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.825+0000 7f73dffff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e0141520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38466/0 (socket says 172.21.15.47:38466) 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 1 -- 172.21.15.47:0/2667144599 learned_addr learned my addr 172.21.15.47:0/2667144599 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): get_auth_request con 0x7f73e013e820 auth_method 0 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): get_auth_request con 0x7f73e0071be0 auth_method 0 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): get_auth_request con 0x7f73e00b0ca0 auth_method 0 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e578d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73e4f8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.826+0000 7f73dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 10 monclient(hunting): handle_auth_done global_id 22342 payload 824 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 -- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f73e0071be0 msgr2=0x7f73e01cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f73e0071be0 0x7f73e01cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 -- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 msgr2=0x7f73e0141520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e0141520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 10 monclient: _finish_hunting 0 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 monclient: found mon.noname-b 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.829+0000 7f73e578d640 1 -- 172.21.15.47:0/2667144599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d0009a70 con 0x7f73e013e820 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.831+0000 7f73e578d640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 secure :-1 s=READY pgs=5513 cs=0 l=1 rev1=1 crypto rx=0x7f73c800c390 tx=0x7f73c800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.831+0000 7f73dd7fa640 10 client.?.objecter ms_handle_connect 0x7f73e013e820 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.831+0000 7f73dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80052d0 con 0x7f73e013e820 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: dump: 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.838 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: _finish_auth 0 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: _check_auth_tickets 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c8005430 con 0x7f73e013e820 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 5 monclient: authenticate success, global_id 22342 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 10 monclient: _renew_subs 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e01cbaa0 con 0x7f73e013e820 2024-03-30T07:44:53.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8003810 con 0x7f73e013e820 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73dd7fa640 10 monclient: dump: 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73deffd640 4 set_mon_vals no callback set 2024-03-30T07:44:53.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 10 client.22342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 10 monclient: _renew_subs 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01cce20 con 0x7f73e013e820 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.835+0000 7f73e71f0640 1 librados: init done 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.836+0000 7f73e71f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.836+0000 7f73e71f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.836+0000 7f73e71f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:53.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.836+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e004e030 con 0x7f73e013e820 2024-03-30T07:44:53.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.854+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f73c8003970 con 0x7f73e013e820 2024-03-30T07:44:53.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.855+0000 7f73dd7fa640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 0x7f73c006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:53.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.855+0000 7f73e4f8c640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 0x7f73c006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:53.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.855+0000 7f73e4f8c640 10 monclient: get_auth_request con 0x7f73c006c590 auth_method 0 2024-03-30T07:44:53.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.855+0000 7f73dd7fa640 10 client.22342.objecter ms_handle_connect 0x7f73c006c590 2024-03-30T07:44:53.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.855+0000 7f73e4f8c640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 0x7f73c006ea50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f73d80011e0 tx=0x7f73d8007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:53.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.857+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f73c8057b80 con 0x7f73e013e820 2024-03-30T07:44:53.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.857+0000 7f73dd7fa640 10 client.22342.objecter ms_dispatch 0x7f73e0000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:53.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.857+0000 7f73dd7fa640 3 client.22342.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:53.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.857+0000 7f73dd7fa640 3 client.22342.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:53.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:53.857+0000 7f73dd7fa640 20 client.22342.objecter dump_active .. 0 homeless 2024-03-30T07:44:54.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.113+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f73c8053ec0 con 0x7f73e013e820 2024-03-30T07:44:54.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.113+0000 7f73dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:54.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.113+0000 7f73dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:54.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.298+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f73e00b37b0 con 0x7f73c006c590 2024-03-30T07:44:54.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.298+0000 7f73dd7fa640 1 -- 172.21.15.47:0/2667144599 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f73d8005850 con 0x7f73c006c590 2024-03-30T07:44:54.300 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 0 to deep-scrub 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 10 librados: watch_flush enter 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 10 librados: watch_flush exit 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 20 client.22342.objecter shutdown clearing up homeless session... 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 10 client.22342.objecter successfully canceled tick 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 msgr2=0x7f73c006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 0x7f73c006ea50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f73d80011e0 tx=0x7f73d8007d30 comp rx=0 tx=0).stop 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 10 monclient: shutdown 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 msgr2=0x7f73e0141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.300+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 secure :-1 s=READY pgs=5513 cs=0 l=1 rev1=1 crypto rx=0x7f73c800c390 tx=0x7f73c800c850 comp rx=0 tx=0).stop 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 shutdown_connections 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f73c006c590 0x7f73c006ea50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f73e013e820 0x7f73e0141a60 unknown :-1 s=CLOSED pgs=5513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f73e00b0ca0 0x7f73e0141520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 --2- 172.21.15.47:0/2667144599 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f73e0071be0 0x7f73e01cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 >> 172.21.15.47:0/2667144599 conn(0x7f73e00b73d0 msgr2=0x7f73e013aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 shutdown_connections 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 -- 172.21.15.47:0/2667144599 wait complete. 2024-03-30T07:44:54.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.301+0000 7f73e71f0640 1 librados: shutdown 2024-03-30T07:44:54.314 INFO:tasks.ceph:Scrubbing osd.1 2024-03-30T07:44:54.314 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:54.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.419+0000 7f1a8421a640 10 monclient: get_monmap_and_config 2024-03-30T07:44:54.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.419+0000 7f1a8421a640 10 monclient: build_initial_monmap 2024-03-30T07:44:54.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.419+0000 7f1a8421a640 10 monclient: monmap: 2024-03-30T07:44:54.421 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:54.421 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:54.420319+0000 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:54.420319+0000 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:54.422 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 Processor -- start 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 -- start start 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient: init 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1a7c0ad7b0 0x7f1a7c141e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c0ad7b0 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c0adb90 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1a7c142380 0x7f1a7c071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c142380 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7c071dc0 con 0x7f1a7c0ad7b0 2024-03-30T07:44:54.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7c13bcf0 con 0x7f1a7c142380 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c06ed90 con 0x7f1a7c0adb90 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.420+0000 7f1a8421a640 10 monclient(hunting): authenticate will time out at 3220.536071s 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a81fb6640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1a7c0ad7b0 0x7f1a7c141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a827b7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1a7c142380 0x7f1a7c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60014/0 (socket says 172.21.15.47:60014) 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 1 -- 172.21.15.47:0/380964087 learned_addr learned my addr 172.21.15.47:0/380964087 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 10 monclient(hunting): get_auth_request con 0x7f1a7c0adb90 auth_method 0 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a817b5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a81fb6640 10 monclient(hunting): get_auth_request con 0x7f1a7c0ad7b0 auth_method 0 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a81fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a81fb6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.421+0000 7f1a81fb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): get_auth_request con 0x7f1a7c142380 auth_method 0 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.422+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 10 monclient(hunting): handle_auth_done global_id 17072 payload 274 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 -- 172.21.15.47:0/380964087 >> v2:172.21.15.52:3300/0 conn(0x7f1a7c142380 msgr2=0x7f1a7c071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 --2- 172.21.15.47:0/380964087 >> v2:172.21.15.52:3300/0 conn(0x7f1a7c142380 0x7f1a7c071880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 -- 172.21.15.47:0/380964087 >> v2:172.21.15.47:3300/0 conn(0x7f1a7c0ad7b0 msgr2=0x7f1a7c141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 --2- 172.21.15.47:0/380964087 >> v2:172.21.15.47:3300/0 conn(0x7f1a7c0ad7b0 0x7f1a7c141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 10 monclient: _finish_hunting 0 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 monclient: found mon.noname-b 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.424+0000 7f1a817b5640 1 -- 172.21.15.47:0/380964087 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c0af640 con 0x7f1a7c0adb90 2024-03-30T07:44:54.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.426+0000 7f1a817b5640 1 --2- 172.21.15.47:0/380964087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c06bff0 tx=0x7f1a6c02e050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6161f47547d17b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 1 -- 172.21.15.47:0/380964087 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c02ee00 con 0x7f1a7c0adb90 2024-03-30T07:44:54.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: dump: 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: _finish_auth 0 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: _check_auth_tickets 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:54.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 1 -- 172.21.15.47:0/380964087 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c036440 con 0x7f1a7c0adb90 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a8421a640 5 monclient: authenticate success, global_id 17072 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a8421a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a8421a640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 1 -- 172.21.15.47:0/380964087 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0367d0 con 0x7f1a7c0adb90 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a8421a640 4 set_mon_vals no callback set 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.429+0000 7f1a80fb4640 10 monclient: dump: 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.433 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.434 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.434 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.434 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.434 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 10 monclient: shutdown 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 -- 172.21.15.47:0/380964087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 msgr2=0x7f1a7c071340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 --2- 172.21.15.47:0/380964087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c06bff0 tx=0x7f1a6c02e050 comp rx=0 tx=0).stop 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 -- 172.21.15.47:0/380964087 shutdown_connections 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 --2- 172.21.15.47:0/380964087 >> v2:172.21.15.52:3300/0 conn(0x7f1a7c142380 0x7f1a7c071880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 --2- 172.21.15.47:0/380964087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c0adb90 0x7f1a7c071340 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 --2- 172.21.15.47:0/380964087 >> v2:172.21.15.47:3300/0 conn(0x7f1a7c0ad7b0 0x7f1a7c141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 -- 172.21.15.47:0/380964087 >> 172.21.15.47:0/380964087 conn(0x7f1a7c0b2f90 msgr2=0x7f1a7c0b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 -- 172.21.15.47:0/380964087 shutdown_connections 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.430+0000 7f1a8421a640 1 -- 172.21.15.47:0/380964087 wait complete. 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.431+0000 7f1a8421a640 10 monclient: build_initial_monmap 2024-03-30T07:44:54.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.431+0000 7f1a8421a640 10 monclient: monmap: 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.431+0000 7f1a8421a640 1 librados: starting msgr at 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.431+0000 7f1a8421a640 1 librados: starting objecter 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.431+0000 7f1a8421a640 1 Processor -- start 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 -- start start 2024-03-30T07:44:54.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 librados: setting wanted keys 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 librados: calling monclient init 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient: init 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a7c0ad7b0 0x7f1a7c137870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c0ad7b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c142380 0x7f1a7c13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c142380 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7c13bfb0 0x7f1a7c13c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c13bfb0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c13b1a0 con 0x7f1a7c0ad7b0 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c13e200 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c138da0 con 0x7f1a7c142380 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.432+0000 7f1a8421a640 10 monclient(hunting): authenticate will time out at 3220.547876s 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c142380 0x7f1a7c13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a827b7640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7c13bfb0 0x7f1a7c13c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c142380 0x7f1a7c13dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60022/0 (socket says 172.21.15.47:60022) 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a81fb6640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a7c0ad7b0 0x7f1a7c137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 1 -- 172.21.15.47:0/2033165255 learned_addr learned my addr 172.21.15.47:0/2033165255 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 10 monclient(hunting): get_auth_request con 0x7f1a7c142380 auth_method 0 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a817b5640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a827b7640 10 monclient(hunting): get_auth_request con 0x7f1a7c13bfb0 auth_method 0 2024-03-30T07:44:54.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a827b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a827b7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a827b7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a81fb6640 10 monclient(hunting): get_auth_request con 0x7f1a7c0ad7b0 auth_method 0 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a81fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a81fb6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.433+0000 7f1a81fb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a827b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a817b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.434+0000 7f1a81fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 10 monclient(hunting): handle_auth_done global_id 22348 payload 824 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 -- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c142380 msgr2=0x7f1a7c13dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1a7c142380 0x7f1a7c13dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 -- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a7c0ad7b0 msgr2=0x7f1a7c137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1a7c0ad7b0 0x7f1a7c137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 10 monclient: _finish_hunting 0 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 monclient: found mon.noname-b 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.437+0000 7f1a827b7640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c00b770 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.439+0000 7f1a827b7640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7c13bfb0 0x7f1a7c13c460 secure :-1 s=READY pgs=5514 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400eb70 tx=0x7f1a7400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.439+0000 7f1a727fc640 10 client.?.objecter ms_handle_connect 0x7f1a7c13bfb0 2024-03-30T07:44:54.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.439+0000 7f1a727fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400cd20 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: dump: 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.447 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: _finish_auth 0 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: _check_auth_tickets 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7400f040 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 5 monclient: authenticate success, global_id 22348 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 10 monclient: _renew_subs 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c139080 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a740146c0 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a727fc640 10 monclient: dump: 2024-03-30T07:44:54.448 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a73fff640 4 set_mon_vals no callback set 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 10 client.22348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 10 monclient: _renew_subs 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:54.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c13c9a0 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.444+0000 7f1a8421a640 1 librados: init done 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 10 client.22348.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 15 client.22348.objecter _session_command_op_assign -1 1 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 20 client.22348.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 10 client.22348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:54.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.445+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a7c141730 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.464+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a74014820 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.465+0000 7f1a727fc640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a6006c5e0 0x7f1a6006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.465+0000 7f1a81fb6640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a6006c5e0 0x7f1a6006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.465+0000 7f1a81fb6640 10 monclient: get_auth_request con 0x7f1a6006c5e0 auth_method 0 2024-03-30T07:44:54.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.465+0000 7f1a81fb6640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a6006c5e0 0x7f1a6006eaa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009ac0 tx=0x7f1a78009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.465+0000 7f1a727fc640 10 client.22348.objecter ms_handle_connect 0x7f1a6006c5e0 2024-03-30T07:44:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f1a740597c0 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 10 client.22348.objecter ms_dispatch 0x7f1a7c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 3 client.22348.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 3 client.22348.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:54.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 10 client.22348.objecter checking command 1 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7f1a60073fb0 0x7f1a60076450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter _get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 15 client.22348.objecter _session_command_op_remove -1 1 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter _get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 15 client.22348.objecter _session_command_op_assign 1 1 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 4 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 10 client.22348.objecter _send_command 1 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a60076b20 con 0x7f1a60073fb0 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 20 client.22348.objecter dump_active .. 0 homeless 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a7400ce80 con 0x7f1a7c13bfb0 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.467+0000 7f1a727fc640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:54.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.468+0000 7f1a817b5640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7f1a60073fb0 0x7f1a60076450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.468+0000 7f1a817b5640 10 monclient: get_auth_request con 0x7f1a60073fb0 auth_method 0 2024-03-30T07:44:54.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.471+0000 7f1a817b5640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7f1a60073fb0 0x7f1a60076450 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.471+0000 7f1a727fc640 10 client.22348.objecter ms_handle_connect 0x7f1a60073fb0 2024-03-30T07:44:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.494+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== osd.1 v2:172.21.15.47:6808/35204 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1a60076b20 con 0x7f1a60073fb0 2024-03-30T07:44:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.494+0000 7f1a727fc640 10 client.22348.objecter ms_dispatch 0x7f1a7c000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:54.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.494+0000 7f1a727fc640 10 client.22348.objecter _finish_command 1 = system:0 2024-03-30T07:44:54.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.494+0000 7f1a727fc640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 4 2024-03-30T07:44:54.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.494+0000 7f1a727fc640 15 client.22348.objecter _session_command_op_remove 1 1 2024-03-30T07:44:54.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 10 client.22348.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 15 client.22348.objecter _session_command_op_assign -1 2 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 20 client.22348.objecter _get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 20 client.22348.objecter _get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 15 client.22348.objecter _session_command_op_remove -1 2 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 20 client.22348.objecter get_session s=0x7f1a600738e0 osd=1 3 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 15 client.22348.objecter _session_command_op_assign 1 2 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 4 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 10 client.22348.objecter _send_command 2 2024-03-30T07:44:54.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.507+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 --> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1a7c13e6d0 con 0x7f1a60073fb0 2024-03-30T07:44:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a727fc640 1 -- 172.21.15.47:0/2033165255 <== osd.1 v2:172.21.15.47:6808/35204 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1a7c13e6d0 con 0x7f1a60073fb0 2024-03-30T07:44:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a727fc640 10 client.22348.objecter ms_dispatch 0x7f1a7c000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a727fc640 10 client.22348.objecter _finish_command 2 = system:0 2024-03-30T07:44:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a727fc640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 4 2024-03-30T07:44:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a727fc640 15 client.22348.objecter _session_command_op_remove 1 2 2024-03-30T07:44:54.523 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:54.523 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:54.523 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 10 librados: watch_flush enter 2024-03-30T07:44:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 10 librados: watch_flush exit 2024-03-30T07:44:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 10 client.22348.objecter close_session for osd.1 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7f1a60073fb0 msgr2=0x7f1a60076450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.47:6808/35204,v1:172.21.15.47:6809/35204] conn(0x7f1a60073fb0 0x7f1a60076450 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 20 client.22348.objecter put_session s=0x7f1a600738e0 osd=1 1 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 20 client.22348.objecter shutdown clearing up homeless session... 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 10 client.22348.objecter successfully canceled tick 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a6006c5e0 msgr2=0x7f1a6006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1a6006c5e0 0x7f1a6006eaa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009ac0 tx=0x7f1a78009190 comp rx=0 tx=0).stop 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 10 monclient: shutdown 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7c13bfb0 msgr2=0x7f1a7c13c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 --2- 172.21.15.47:0/2033165255 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1a7c13bfb0 0x7f1a7c13c460 secure :-1 s=READY pgs=5514 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400eb70 tx=0x7f1a7400c490 comp rx=0 tx=0).stop 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a827b7640 1 -- 172.21.15.47:0/2033165255 reap_dead start 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 shutdown_connections 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.520+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 >> 172.21.15.47:0/2033165255 conn(0x7f1a7c0b2f90 msgr2=0x7f1a7c13f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.521+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 shutdown_connections 2024-03-30T07:44:54.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.521+0000 7f1a8421a640 1 -- 172.21.15.47:0/2033165255 wait complete. 2024-03-30T07:44:54.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.521+0000 7f1a8421a640 1 librados: shutdown 2024-03-30T07:44:54.532 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.633+0000 7f4067caa640 10 monclient: get_monmap_and_config 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.633+0000 7f4067caa640 10 monclient: build_initial_monmap 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.633+0000 7f4067caa640 10 monclient: monmap: 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:54.634012+0000 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:54.634012+0000 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.635 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:54.636 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:54.636 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:54.636 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:54.636 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:54.636 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.633+0000 7f4067caa640 1 Processor -- start 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 1 -- start start 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient: init 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 0x7f40600a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-b con 0x7f40600b4280 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f406013e930 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f406013ed70 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600aa3c0 con 0x7f406013e930 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600aa630 con 0x7f406013ed70 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600b4660 con 0x7f40600b4280 2024-03-30T07:44:54.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.634+0000 7f4067caa640 10 monclient(hunting): authenticate will time out at 3220.749829s 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065a46640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 0x7f40600a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065245640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f406013e930 0x7f4060141670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55254/0 (socket says 172.21.15.47:55254) 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065245640 1 -- 172.21.15.47:0/856307155 learned_addr learned my addr 172.21.15.47:0/856307155 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065a46640 10 monclient(hunting): get_auth_request con 0x7f40600b4280 auth_method 0 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065a46640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.635+0000 7f4065a46640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.636+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.636+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.636+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 10 monclient(hunting): handle_auth_done global_id 17078 payload 274 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 1 -- 172.21.15.47:0/856307155 >> v1:172.21.15.52:6789/0 conn(0x7f406013ed70 legacy=0x7f4060141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 1 -- 172.21.15.47:0/856307155 >> v1:172.21.15.47:6789/0 conn(0x7f406013e930 legacy=0x7f4060141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 10 monclient: _finish_hunting 0 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 1 monclient: found mon.noname-b 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.637+0000 7f4065a46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.638+0000 7f4065a46640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:54.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.638+0000 7f4065a46640 1 -- 172.21.15.47:0/856307155 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40600a9210 con 0x7f40600b4280 2024-03-30T07:44:54.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.640+0000 7f4065a46640 1 --2- 172.21.15.47:0/856307155 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 0x7f40600a8990 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f40580097d0 tx=0x7f405802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b551ab5a344d9ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 1 -- 172.21.15.47:0/856307155 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058031800 con 0x7f40600b4280 2024-03-30T07:44:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:54.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: dump: 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: _finish_auth 0 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: _check_auth_tickets 2024-03-30T07:44:54.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 1 -- 172.21.15.47:0/856307155 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058031960 con 0x7f40600b4280 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4067caa640 5 monclient: authenticate success, global_id 17078 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4067caa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4067caa640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 1 -- 172.21.15.47:0/856307155 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058031cf0 con 0x7f40600b4280 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4067caa640 4 set_mon_vals no callback set 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.643+0000 7f4064a44640 10 monclient: dump: 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.647 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.648 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 10 monclient: shutdown 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 -- 172.21.15.47:0/856307155 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 msgr2=0x7f40600a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 --2- 172.21.15.47:0/856307155 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 0x7f40600a8990 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f40580097d0 tx=0x7f405802b8a0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 -- 172.21.15.47:0/856307155 shutdown_connections 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 --2- 172.21.15.47:0/856307155 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f40600b4280 0x7f40600a8990 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 -- 172.21.15.47:0/856307155 >> 172.21.15.47:0/856307155 conn(0x7f4060065e00 msgr2=0x7f4060066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 -- 172.21.15.47:0/856307155 shutdown_connections 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.644+0000 7f4067caa640 1 -- 172.21.15.47:0/856307155 wait complete. 2024-03-30T07:44:54.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 10 monclient: build_initial_monmap 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 10 monclient: monmap: 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 librados: starting msgr at 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 librados: starting objecter 2024-03-30T07:44:54.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 Processor -- start 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 -- start start 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 librados: setting wanted keys 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 1 librados: calling monclient init 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.645+0000 7f4067caa640 10 monclient: init 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-a con 0x7f40600b4280 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f406013e930 0x7f40601455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-b con 0x7f406013e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f406013ed70 0x7f4060142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): picked mon.noname-c con 0x7f406013ed70 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4060145180 con 0x7f40600b4280 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4060144fa0 con 0x7f406013e930 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40601448a0 con 0x7f406013ed70 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4067caa640 10 monclient(hunting): authenticate will time out at 3220.761542s 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.646+0000 7f4065a46640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:33162/0 (socket says 172.21.15.47:33162) 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 1 -- 172.21.15.47:0/516419604 learned_addr learned my addr 172.21.15.47:0/516419604 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 10 monclient(hunting): get_auth_request con 0x7f40600b4280 auth_method 0 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065a46640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.647+0000 7f4065245640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f406013e930 0x7f40601455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065245640 10 monclient(hunting): get_auth_request con 0x7f406013e930 auth_method 0 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065245640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065245640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.648+0000 7f4065a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.649+0000 7f4065245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.649+0000 7f4065245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.649+0000 7f4065245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.649+0000 7f4066247640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f406013ed70 0x7f4060142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): get_auth_request con 0x7f406013ed70 auth_method 0 2024-03-30T07:44:54.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.650+0000 7f4066247640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 10 monclient(hunting): handle_auth_done global_id 15858 payload 824 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 -- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f406013ed70 msgr2=0x7f4060142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f406013ed70 0x7f4060142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 -- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f406013e930 msgr2=0x7f40601455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f406013e930 0x7f40601455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 10 monclient: _finish_hunting 0 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 monclient: found mon.noname-a 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:54.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:54.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.651+0000 7f4065a46640 1 -- 172.21.15.47:0/516419604 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058009400 con 0x7f40600b4280 2024-03-30T07:44:54.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.654+0000 7f4065a46640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f40580098f0 tx=0x7f40580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.654+0000 7f405e7fc640 10 client.?.objecter ms_handle_connect 0x7f40600b4280 2024-03-30T07:44:54.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.654+0000 7f405e7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405803bcc0 con 0x7f40600b4280 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: dump: 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: _finish_auth 0 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405803be60 con 0x7f40600b4280 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f4067caa640 5 monclient: authenticate success, global_id 15858 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f4067caa640 10 monclient: _renew_subs 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f4067caa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4060144b80 con 0x7f40600b4280 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058045630 con 0x7f40600b4280 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405e7fc640 10 monclient: dump: 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.709+0000 7f405ffff640 4 set_mon_vals no callback set 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.710+0000 7f4067caa640 10 client.15858.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.710+0000 7f4067caa640 10 monclient: _renew_subs 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.710+0000 7f4067caa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.710+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4060142eb0 con 0x7f40600b4280 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.710+0000 7f4067caa640 1 librados: init done 2024-03-30T07:44:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.711+0000 7f4067caa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:54.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.711+0000 7f4067caa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:54.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.711+0000 7f4067caa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:54.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.711+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406004e030 con 0x7f40600b4280 2024-03-30T07:44:54.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.982+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f40580443a0 con 0x7f40600b4280 2024-03-30T07:44:54.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.982+0000 7f405e7fc640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 0x7f404c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:54.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.982+0000 7f4065245640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 0x7f404c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:54.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.982+0000 7f4065245640 10 monclient: get_auth_request con 0x7f404c06c5e0 auth_method 0 2024-03-30T07:44:54.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.983+0000 7f4065245640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 0x7f404c06eaa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4060145be0 tx=0x7f4054009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:54.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.983+0000 7f405e7fc640 10 client.15858.objecter ms_handle_connect 0x7f404c06c5e0 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f405802e070 con 0x7f40600b4280 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 10 client.15858.objecter ms_dispatch 0x7f4060000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 3 client.15858.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 3 client.15858.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 20 client.15858.objecter dump_active .. 0 homeless 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4058077120 con 0x7f40600b4280 2024-03-30T07:44:54.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:54.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:54.995+0000 7f405e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:55.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.190+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f40600a8ed0 con 0x7f404c06c5e0 2024-03-30T07:44:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.191+0000 7f405e7fc640 1 -- 172.21.15.47:0/516419604 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f405400f820 con 0x7f404c06c5e0 2024-03-30T07:44:55.193 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 1 to deep-scrub 2024-03-30T07:44:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 10 librados: watch_flush enter 2024-03-30T07:44:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 10 librados: watch_flush exit 2024-03-30T07:44:55.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 20 client.15858.objecter shutdown clearing up homeless session... 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 10 client.15858.objecter successfully canceled tick 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 msgr2=0x7f404c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 0x7f404c06eaa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4060145be0 tx=0x7f4054009040 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 10 monclient: shutdown 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 msgr2=0x7f4060134600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f40580098f0 tx=0x7f40580046c0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 shutdown_connections 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f404c06c5e0 0x7f404c06eaa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f406013ed70 0x7f4060142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f406013e930 0x7f40601455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 --2- 172.21.15.47:0/516419604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f40600b4280 0x7f4060134600 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 >> 172.21.15.47:0/516419604 conn(0x7f4060065e00 msgr2=0x7f40600b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 shutdown_connections 2024-03-30T07:44:55.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 -- 172.21.15.47:0/516419604 wait complete. 2024-03-30T07:44:55.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.193+0000 7f4067caa640 1 librados: shutdown 2024-03-30T07:44:55.206 INFO:tasks.ceph:Scrubbing osd.2 2024-03-30T07:44:55.206 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:55.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.308+0000 7fb44172e640 10 monclient: get_monmap_and_config 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.308+0000 7fb44172e640 10 monclient: build_initial_monmap 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.308+0000 7fb44172e640 10 monclient: monmap: 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:55.309662+0000 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:55.309662+0000 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:55.311 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.309+0000 7fb44172e640 1 Processor -- start 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.309+0000 7fb44172e640 1 -- start start 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.309+0000 7fb44172e640 10 monclient: init 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb43c06a540 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fb43c0b04f0 0x7fb43c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-c con 0x7fb43c0b04f0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 0x7fb43c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-b con 0x7fb43c0707a0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb43c06acd0 con 0x7fb43c06a540 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43c0b0980 con 0x7fb43c0b04f0 2024-03-30T07:44:55.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c06b9f0 con 0x7fb43c0707a0 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb44172e640 10 monclient(hunting): authenticate will time out at 3221.425278s 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb43b7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 0x7fb43c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb43affd640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb43c06a540 0x7fb43c0afde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55266/0 (socket says 172.21.15.47:55266) 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb43affd640 1 -- 172.21.15.47:0/685610729 learned_addr learned my addr 172.21.15.47:0/685610729 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.310+0000 7fb43a7fc640 1 --2- 172.21.15.47:0/685610729 >> v2:172.21.15.52:3300/0 conn(0x7fb43c0b04f0 0x7fb43c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): get_auth_request con 0x7fb43c0707a0 auth_method 0 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): get_auth_request con 0x7fb43c0b04f0 auth_method 0 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.311+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_done global_id 17084 payload 274 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 1 -- 172.21.15.47:0/685610729 >> v2:172.21.15.52:3300/0 conn(0x7fb43c0b04f0 msgr2=0x7fb43c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 1 --2- 172.21.15.47:0/685610729 >> v2:172.21.15.52:3300/0 conn(0x7fb43c0b04f0 0x7fb43c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 1 -- 172.21.15.47:0/685610729 >> v1:172.21.15.47:6789/0 conn(0x7fb43c06a540 legacy=0x7fb43c0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 1 monclient: found mon.noname-b 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.313+0000 7fb43b7fe640 1 -- 172.21.15.47:0/685610729 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43c06f880 con 0x7fb43c0707a0 2024-03-30T07:44:55.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.315+0000 7fb43b7fe640 1 --2- 172.21.15.47:0/685610729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 0x7fb43c070ba0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009d90 tx=0x7fb42c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb4d30ea84986bd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 1 -- 172.21.15.47:0/685610729 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c02cbc0 con 0x7fb43c0707a0 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: dump: 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.321 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: _finish_auth 0 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 1 -- 172.21.15.47:0/685610729 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c02cd20 con 0x7fb43c0707a0 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 1 -- 172.21.15.47:0/685610729 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42c034660 con 0x7fb43c0707a0 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 5 monclient: authenticate success, global_id 17084 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 4 set_mon_vals no callback set 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:55.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb439ffb640 10 monclient: dump: 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.323 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 10 monclient: shutdown 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 -- 172.21.15.47:0/685610729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 msgr2=0x7fb43c070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 --2- 172.21.15.47:0/685610729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 0x7fb43c070ba0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009d90 tx=0x7fb42c02be10 comp rx=0 tx=0).stop 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 -- 172.21.15.47:0/685610729 shutdown_connections 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 --2- 172.21.15.47:0/685610729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c0707a0 0x7fb43c070ba0 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 --2- 172.21.15.47:0/685610729 >> v2:172.21.15.52:3300/0 conn(0x7fb43c0b04f0 0x7fb43c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 -- 172.21.15.47:0/685610729 >> 172.21.15.47:0/685610729 conn(0x7fb43c0b5100 msgr2=0x7fb43c0b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 -- 172.21.15.47:0/685610729 shutdown_connections 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.319+0000 7fb44172e640 1 -- 172.21.15.47:0/685610729 wait complete. 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.320+0000 7fb44172e640 10 monclient: build_initial_monmap 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.320+0000 7fb44172e640 10 monclient: monmap: 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:55.327 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.320+0000 7fb44172e640 1 librados: starting msgr at 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.320+0000 7fb44172e640 1 librados: starting objecter 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 Processor -- start 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 -- start start 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 librados: setting wanted keys 2024-03-30T07:44:55.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 librados: calling monclient init 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient: init 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 0x7fb43c0b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-c con 0x7fb43c06a540 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb43c0707a0 0x7fb43c1a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-b con 0x7fb43c0707a0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0b04f0 0x7fb43c1d6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): picked mon.noname-a con 0x7fb43c0b04f0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c1a6e40 con 0x7fb43c0b04f0 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c1a6160 con 0x7fb43c0707a0 2024-03-30T07:44:55.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43c1d67b0 con 0x7fb43c06a540 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.321+0000 7fb44172e640 10 monclient(hunting): authenticate will time out at 3221.436895s 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 0x7fb43c0b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43b7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0b04f0 0x7fb43c1d6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 0x7fb43c0b2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60062/0 (socket says 172.21.15.47:60062) 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 1 -- 172.21.15.47:0/503731538 learned_addr learned my addr 172.21.15.47:0/503731538 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:55.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): get_auth_request con 0x7fb43c06a540 auth_method 0 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43a7fc640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb43c0707a0 0x7fb43c1a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43b7fe640 10 monclient(hunting): get_auth_request con 0x7fb43c0b04f0 auth_method 0 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43a7fc640 10 monclient(hunting): get_auth_request con 0x7fb43c0707a0 auth_method 0 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.322+0000 7fb43affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.323+0000 7fb43b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 10 monclient(hunting): handle_auth_done global_id 22360 payload 824 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 -- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 msgr2=0x7fb43c0b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 0x7fb43c0b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 -- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0b04f0 msgr2=0x7fb43c1d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb43c0b04f0 0x7fb43c1d6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 10 monclient: _finish_hunting 0 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 monclient: found mon.noname-b 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43a7fc640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42c009a70 con 0x7fb43c0707a0 2024-03-30T07:44:55.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.326+0000 7fb43affd640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb43c06a540 0x7fb43c0b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:55.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.328+0000 7fb43a7fc640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb43c0707a0 0x7fb43c1a5c20 secure :-1 s=READY pgs=5516 cs=0 l=1 rev1=1 crypto rx=0x7fb42400e870 tx=0x7fb42400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.328+0000 7fb4337fe640 10 client.?.objecter ms_handle_connect 0x7fb43c0707a0 2024-03-30T07:44:55.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.328+0000 7fb4337fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400ee60 con 0x7fb43c0707a0 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: dump: 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.335 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42400c940 con 0x7fb43c0707a0 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42400ccd0 con 0x7fb43c0707a0 2024-03-30T07:44:55.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 5 monclient: authenticate success, global_id 22360 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 10 monclient: _renew_subs 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43c1d72d0 con 0x7fb43c0707a0 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4397fa640 4 set_mon_vals no callback set 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb4337fe640 10 monclient: dump: 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.337 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 10 client.22360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 10 monclient: _renew_subs 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43c1c9050 con 0x7fb43c0707a0 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.332+0000 7fb44172e640 1 librados: init done 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 10 client.22360.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 15 client.22360.objecter _session_command_op_assign -1 1 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 20 client.22360.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 10 client.22360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.333+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb43c06a920 con 0x7fb43c0707a0 2024-03-30T07:44:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.353+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb424013470 con 0x7fb43c0707a0 2024-03-30T07:44:55.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.353+0000 7fb4337fe640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb41c06c5c0 0x7fb41c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.354+0000 7fb43affd640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb41c06c5c0 0x7fb41c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.354+0000 7fb43affd640 10 monclient: get_auth_request con 0x7fb41c06c5c0 auth_method 0 2024-03-30T07:44:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.354+0000 7fb43affd640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb41c06c5c0 0x7fb41c06ea80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb434005880 tx=0x7fb434009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.354+0000 7fb4337fe640 10 client.22360.objecter ms_handle_connect 0x7fb41c06c5c0 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.355+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fb42400ce30 con 0x7fb43c0707a0 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.355+0000 7fb4337fe640 10 client.22360.objecter ms_dispatch 0x7fb43c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.355+0000 7fb4337fe640 3 client.22360.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.355+0000 7fb4337fe640 3 client.22360.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 10 client.22360.objecter checking command 1 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fb41c073f90 0x7fb41c076430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter _get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 15 client.22360.objecter _session_command_op_remove -1 1 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter _get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 15 client.22360.objecter _session_command_op_assign 2 1 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 4 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 10 client.22360.objecter _send_command 1 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb41c076b00 con 0x7fb41c073f90 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 20 client.22360.objecter dump_active .. 0 homeless 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb424054730 con 0x7fb43c0707a0 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb4337fe640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:55.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.356+0000 7fb43b7fe640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fb41c073f90 0x7fb41c076430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.357+0000 7fb43b7fe640 10 monclient: get_auth_request con 0x7fb41c073f90 auth_method 0 2024-03-30T07:44:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.359+0000 7fb4337fe640 10 client.22360.objecter ms_handle_connect 0x7fb41c073f90 2024-03-30T07:44:55.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.359+0000 7fb43b7fe640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fb41c073f90 0x7fb41c076430 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.383+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== osd.2 v2:172.21.15.47:6824/35207 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb41c076b00 con 0x7fb41c073f90 2024-03-30T07:44:55.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.383+0000 7fb4337fe640 10 client.22360.objecter ms_dispatch 0x7fb43c000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:55.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.383+0000 7fb4337fe640 10 client.22360.objecter _finish_command 1 = system:0 2024-03-30T07:44:55.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.383+0000 7fb4337fe640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 4 2024-03-30T07:44:55.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.383+0000 7fb4337fe640 15 client.22360.objecter _session_command_op_remove 2 1 2024-03-30T07:44:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 10 client.22360.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:55.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 15 client.22360.objecter _session_command_op_assign -1 2 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 20 client.22360.objecter _get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 20 client.22360.objecter _get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 15 client.22360.objecter _session_command_op_remove -1 2 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 20 client.22360.objecter get_session s=0x7fb41c0738c0 osd=2 3 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 15 client.22360.objecter _session_command_op_assign 2 2 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 4 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 10 client.22360.objecter _send_command 2 2024-03-30T07:44:55.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.395+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 --> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb43c06fe40 con 0x7fb41c073f90 2024-03-30T07:44:55.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.411+0000 7fb4337fe640 1 -- 172.21.15.47:0/503731538 <== osd.2 v2:172.21.15.47:6824/35207 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb42c02ce90 con 0x7fb41c073f90 2024-03-30T07:44:55.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.411+0000 7fb4337fe640 10 client.22360.objecter ms_dispatch 0x7fb43c000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:55.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.411+0000 7fb4337fe640 10 client.22360.objecter _finish_command 2 = system:0 2024-03-30T07:44:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.411+0000 7fb4337fe640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 4 2024-03-30T07:44:55.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.411+0000 7fb4337fe640 15 client.22360.objecter _session_command_op_remove 2 2 2024-03-30T07:44:55.414 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:55.414 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:55.414 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:55.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 10 librados: watch_flush enter 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 10 librados: watch_flush exit 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 10 client.22360.objecter close_session for osd.2 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fb41c073f90 msgr2=0x7fb41c076430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.47:6824/35207,v1:172.21.15.47:6825/35207] conn(0x7fb41c073f90 0x7fb41c076430 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 20 client.22360.objecter put_session s=0x7fb41c0738c0 osd=2 1 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 20 client.22360.objecter shutdown clearing up homeless session... 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 10 client.22360.objecter successfully canceled tick 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb41c06c5c0 msgr2=0x7fb41c06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fb41c06c5c0 0x7fb41c06ea80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb434005880 tx=0x7fb434009040 comp rx=0 tx=0).stop 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 10 monclient: shutdown 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb43c0707a0 msgr2=0x7fb43c1a5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 --2- 172.21.15.47:0/503731538 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fb43c0707a0 0x7fb43c1a5c20 secure :-1 s=READY pgs=5516 cs=0 l=1 rev1=1 crypto rx=0x7fb42400e870 tx=0x7fb42400ed30 comp rx=0 tx=0).stop 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb43b7fe640 1 -- 172.21.15.47:0/503731538 reap_dead start 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 shutdown_connections 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 >> 172.21.15.47:0/503731538 conn(0x7fb43c0b5100 msgr2=0x7fb43c0ade00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 shutdown_connections 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 -- 172.21.15.47:0/503731538 wait complete. 2024-03-30T07:44:55.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.412+0000 7fb44172e640 1 librados: shutdown 2024-03-30T07:44:55.424 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.523+0000 7f1e72c3a640 10 monclient: get_monmap_and_config 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.523+0000 7f1e72c3a640 10 monclient: build_initial_monmap 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.523+0000 7f1e72c3a640 10 monclient: monmap: 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.525 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:55.524282+0000 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:55.524282+0000 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:55.526 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 Processor -- start 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 -- start start 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient: init 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e6c0b4280 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c13ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c13e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 0x7f1e6c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c141670 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c13f2b0 con 0x7f1e6c0b4280 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6c0b4660 con 0x7f1e6c141670 2024-03-30T07:44:55.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c0aa3c0 con 0x7f1e6c13e930 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.524+0000 7f1e72c3a640 10 monclient(hunting): authenticate will time out at 3221.640078s 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e711d7640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 0x7f1e6c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e6bfff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c13ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e709d6640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1e6c0b4280 0x7f1e6c0a8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:55274/0 (socket says 172.21.15.47:55274) 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e709d6640 1 -- 172.21.15.47:0/2383166589 learned_addr learned my addr 172.21.15.47:0/2383166589 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e711d7640 10 monclient(hunting): get_auth_request con 0x7f1e6c141670 auth_method 0 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e711d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e711d7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e711d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request con 0x7f1e6c13e930 auth_method 0 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.525+0000 7f1e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.526+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_done global_id 17090 payload 274 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 1 -- 172.21.15.47:0/2383166589 >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 msgr2=0x7f1e6c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 1 --2- 172.21.15.47:0/2383166589 >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 0x7f1e6c141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 1 -- 172.21.15.47:0/2383166589 >> v1:172.21.15.47:6789/0 conn(0x7f1e6c0b4280 legacy=0x7f1e6c0a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 10 monclient: _finish_hunting 0 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 1 monclient: found mon.noname-b 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:55.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e6bfff640 1 -- 172.21.15.47:0/2383166589 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0aa630 con 0x7f1e6c13e930 2024-03-30T07:44:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.528+0000 7f1e711d7640 1 --2- 172.21.15.47:0/2383166589 >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 0x7f1e6c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:55.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.530+0000 7f1e6bfff640 1 --2- 172.21.15.47:0/2383166589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c13ed70 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097a0 tx=0x7f1e5802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cff48b8043d4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 1 -- 172.21.15.47:0/2383166589 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5802e070 con 0x7f1e6c13e930 2024-03-30T07:44:55.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: dump: 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:55.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 1 -- 172.21.15.47:0/2383166589 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e580315a0 con 0x7f1e6c13e930 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 1 -- 172.21.15.47:0/2383166589 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031930 con 0x7f1e6c13e930 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e6b7fe640 10 monclient: dump: 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.537 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.538 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e72c3a640 5 monclient: authenticate success, global_id 17090 2024-03-30T07:44:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e72c3a640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:55.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.533+0000 7f1e72c3a640 4 set_mon_vals no callback set 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 10 monclient: shutdown 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 -- 172.21.15.47:0/2383166589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 msgr2=0x7f1e6c13ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/2383166589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c13ed70 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097a0 tx=0x7f1e5802b840 comp rx=0 tx=0).stop 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 -- 172.21.15.47:0/2383166589 shutdown_connections 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/2383166589 >> v2:172.21.15.52:3300/0 conn(0x7f1e6c141670 0x7f1e6c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/2383166589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c13ed70 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 -- 172.21.15.47:0/2383166589 >> 172.21.15.47:0/2383166589 conn(0x7f1e6c065e00 msgr2=0x7f1e6c066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:55.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 -- 172.21.15.47:0/2383166589 shutdown_connections 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.534+0000 7f1e72c3a640 1 -- 172.21.15.47:0/2383166589 wait complete. 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 10 monclient: build_initial_monmap 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 10 monclient: monmap: 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 librados: starting msgr at 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 librados: starting objecter 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 Processor -- start 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 -- start start 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 librados: setting wanted keys 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 1 librados: calling monclient init 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.535+0000 7f1e72c3a640 10 monclient: init 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 0x7f1e6c13da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c0b4280 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c13e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e6c141670 0x7f1e6c144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c141670 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c137e40 con 0x7f1e6c141670 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c138040 con 0x7f1e6c0b4280 2024-03-30T07:44:55.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c138320 con 0x7f1e6c13e930 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.536+0000 7f1e72c3a640 10 monclient(hunting): authenticate will time out at 3221.651705s 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 0x7f1e6c13da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e711d7640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e6c141670 0x7f1e6c144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60078/0 (socket says 172.21.15.47:60078) 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 1 -- 172.21.15.47:0/3848615830 learned_addr learned my addr 172.21.15.47:0/3848615830 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request con 0x7f1e6c13e930 auth_method 0 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): get_auth_request con 0x7f1e6c0b4280 auth_method 0 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e711d7640 10 monclient(hunting): get_auth_request con 0x7f1e6c141670 auth_method 0 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e711d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e711d7640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e711d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.537+0000 7f1e709d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.538+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:55.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.538+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:55.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.538+0000 7f1e711d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 10 monclient(hunting): handle_auth_done global_id 22366 payload 824 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 -- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 msgr2=0x7f1e6c1443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 -- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e6c141670 msgr2=0x7f1e6c144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e6c141670 0x7f1e6c144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 10 monclient: _finish_hunting 0 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 monclient: found mon.noname-b 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e709d6640 1 -- 172.21.15.47:0/3848615830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e64009a70 con 0x7f1e6c0b4280 2024-03-30T07:44:55.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.540+0000 7f1e6bfff640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.542+0000 7f1e709d6640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 0x7f1e6c13da20 secure :-1 s=READY pgs=5517 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000c370 tx=0x7f1e6000c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.542+0000 7f1e697fa640 10 client.?.objecter ms_handle_connect 0x7f1e6c0b4280 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.542+0000 7f1e697fa640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005ce0 con 0x7f1e6c0b4280 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: dump: 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.551 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: _finish_auth 0 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: _check_auth_tickets 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60005e40 con 0x7f1e6c0b4280 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 5 monclient: authenticate success, global_id 22366 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 10 monclient: _renew_subs 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e58009400 con 0x7f1e6c0b4280 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60007310 con 0x7f1e6c0b4280 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:55.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e697fa640 10 monclient: dump: 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e6affd640 4 set_mon_vals no callback set 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 10 client.22366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:55.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 10 monclient: _renew_subs 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6c1d9e80 con 0x7f1e6c0b4280 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.546+0000 7f1e72c3a640 1 librados: init done 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.547+0000 7f1e72c3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.547+0000 7f1e72c3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.547+0000 7f1e72c3a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:55.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.547+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6c144cd0 con 0x7f1e6c0b4280 2024-03-30T07:44:55.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.814+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1e60007470 con 0x7f1e6c0b4280 2024-03-30T07:44:55.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.815+0000 7f1e697fa640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 0x7f1e5006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:55.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.815+0000 7f1e6bfff640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 0x7f1e5006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:55.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.815+0000 7f1e6bfff640 10 monclient: get_auth_request con 0x7f1e5006c5e0 auth_method 0 2024-03-30T07:44:55.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.815+0000 7f1e6bfff640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 0x7f1e5006eaa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1e58000c00 tx=0x7f1e5803d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:55.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.816+0000 7f1e697fa640 10 client.22366.objecter ms_handle_connect 0x7f1e5006c5e0 2024-03-30T07:44:55.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.818+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f1e60007720 con 0x7f1e6c0b4280 2024-03-30T07:44:55.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.818+0000 7f1e697fa640 10 client.22366.objecter ms_dispatch 0x7f1e6c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:55.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.818+0000 7f1e697fa640 3 client.22366.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:55.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.818+0000 7f1e697fa640 3 client.22366.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:55.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.818+0000 7f1e697fa640 20 client.22366.objecter dump_active .. 0 homeless 2024-03-30T07:44:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.828+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1e60053cf0 con 0x7f1e6c0b4280 2024-03-30T07:44:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.828+0000 7f1e697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:55.828+0000 7f1e697fa640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:56.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.017+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e6c144cd0 con 0x7f1e5006c5e0 2024-03-30T07:44:56.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.017+0000 7f1e697fa640 1 -- 172.21.15.47:0/3848615830 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e6c144cd0 con 0x7f1e5006c5e0 2024-03-30T07:44:56.020 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 2 to deep-scrub 2024-03-30T07:44:56.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 10 librados: watch_flush enter 2024-03-30T07:44:56.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 10 librados: watch_flush exit 2024-03-30T07:44:56.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 20 client.22366.objecter shutdown clearing up homeless session... 2024-03-30T07:44:56.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 10 client.22366.objecter successfully canceled tick 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 msgr2=0x7f1e5006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 0x7f1e5006eaa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1e58000c00 tx=0x7f1e5803d040 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 10 monclient: shutdown 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 msgr2=0x7f1e6c13da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.019+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 0x7f1e6c13da20 secure :-1 s=READY pgs=5517 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000c370 tx=0x7f1e6000c770 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 shutdown_connections 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f1e5006c5e0 0x7f1e5006eaa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e6c141670 0x7f1e6c144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e6c13e930 0x7f1e6c1443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 --2- 172.21.15.47:0/3848615830 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f1e6c0b4280 0x7f1e6c13da20 unknown :-1 s=CLOSED pgs=5517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 >> 172.21.15.47:0/3848615830 conn(0x7f1e6c065e00 msgr2=0x7f1e6c136c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 shutdown_connections 2024-03-30T07:44:56.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 -- 172.21.15.47:0/3848615830 wait complete. 2024-03-30T07:44:56.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.020+0000 7f1e72c3a640 1 librados: shutdown 2024-03-30T07:44:56.033 INFO:tasks.ceph:Scrubbing osd.3 2024-03-30T07:44:56.033 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 10 monclient: get_monmap_and_config 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 10 monclient: monmap: 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:56.138832+0000 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:56.138832+0000 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:56.140 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:56.141 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 1 Processor -- start 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 1 -- start start 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.138+0000 7f4c75699640 10 monclient: init 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4c700b43c0 0x7f4c700a8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-c con 0x7f4c700b43c0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4c7013e890 0x7f4c7013ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-a con 0x7f4c7013e890 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c70144b80 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7013f210 con 0x7f4c70144b80 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c700a8da0 con 0x7f4c7013e890 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c700b47a0 con 0x7f4c700b43c0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c75699640 10 monclient(hunting): authenticate will time out at 3222.254449s 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c6e7fc640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4c7013e890 0x7f4c7013ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.139+0000 7f4c6effd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4c700b43c0 0x7f4c700a8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4c700b43c0 0x7f4c700a8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38542/0 (socket says 172.21.15.47:38542) 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 1 -- 172.21.15.47:0/793045341 learned_addr learned my addr 172.21.15.47:0/793045341 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): get_auth_request con 0x7f4c700b43c0 auth_method 0 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4c7013e890 auth_method 0 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.140+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.141+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.141+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.141+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 10 monclient(hunting): handle_auth_done global_id 22369 payload 274 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 1 -- 172.21.15.47:0/793045341 >> v2:172.21.15.47:3300/0 conn(0x7f4c7013e890 msgr2=0x7f4c7013ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 1 --2- 172.21.15.47:0/793045341 >> v2:172.21.15.47:3300/0 conn(0x7f4c7013e890 0x7f4c7013ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 1 -- 172.21.15.47:0/793045341 >> v1:172.21.15.47:6789/0 conn(0x7f4c70144b80 legacy=0x7f4c70144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 1 monclient: found mon.noname-c 2024-03-30T07:44:56.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.142+0000 7f4c6effd640 1 -- 172.21.15.47:0/793045341 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c700aa3f0 con 0x7f4c700b43c0 2024-03-30T07:44:56.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.144+0000 7f4c6effd640 1 --2- 172.21.15.47:0/793045341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c700a8860 secure :-1 s=READY pgs=5518 cs=0 l=1 rev1=1 crypto rx=0x7f4c700b81c0 tx=0x7f4c6802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcb830d2ce3ce4c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 1 -- 172.21.15.47:0/793045341 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68031870 con 0x7f4c700b43c0 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: dump: 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.150 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 1 -- 172.21.15.47:0/793045341 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c680319d0 con 0x7f4c700b43c0 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 1 -- 172.21.15.47:0/793045341 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68031d60 con 0x7f4c700b43c0 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 5 monclient: authenticate success, global_id 22369 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 4 set_mon_vals no callback set 2024-03-30T07:44:56.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c6dffb640 10 monclient: dump: 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.152 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 10 monclient: shutdown 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 -- 172.21.15.47:0/793045341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 msgr2=0x7f4c700a8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 --2- 172.21.15.47:0/793045341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c700a8860 secure :-1 s=READY pgs=5518 cs=0 l=1 rev1=1 crypto rx=0x7f4c700b81c0 tx=0x7f4c6802b910 comp rx=0 tx=0).stop 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 -- 172.21.15.47:0/793045341 shutdown_connections 2024-03-30T07:44:56.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 --2- 172.21.15.47:0/793045341 >> v2:172.21.15.47:3300/0 conn(0x7f4c7013e890 0x7f4c7013ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 --2- 172.21.15.47:0/793045341 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c700a8860 unknown :-1 s=CLOSED pgs=5518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 -- 172.21.15.47:0/793045341 >> 172.21.15.47:0/793045341 conn(0x7f4c70071930 msgr2=0x7f4c70071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.148+0000 7f4c75699640 1 -- 172.21.15.47:0/793045341 shutdown_connections 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.149+0000 7f4c75699640 1 -- 172.21.15.47:0/793045341 wait complete. 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.149+0000 7f4c75699640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.149+0000 7f4c75699640 10 monclient: monmap: 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.157 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.149+0000 7f4c75699640 1 librados: starting msgr at 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.149+0000 7f4c75699640 1 librados: starting objecter 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 Processor -- start 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 -- start start 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 librados: setting wanted keys 2024-03-30T07:44:56.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 librados: calling monclient init 2024-03-30T07:44:56.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient: init 2024-03-30T07:44:56.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:44:56.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c701377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-b con 0x7f4c700b43c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:56.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c7013e890 0x7f4c70142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-a con 0x7f4c7013e890 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:56.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4c70144b80 0x7f4c701d9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): picked mon.noname-c con 0x7f4c70144b80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c701d9aa0 con 0x7f4c7013e890 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c70142680 con 0x7f4c700b43c0 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7013f400 con 0x7f4c70144b80 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.150+0000 7f4c75699640 10 monclient(hunting): authenticate will time out at 3222.265841s 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6e7fc640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c7013e890 0x7f4c70142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6f7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4c70144b80 0x7f4c701d9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c701377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c701377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:38558/0 (socket says 172.21.15.47:38558) 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 1 -- 172.21.15.47:0/581019479 learned_addr learned my addr 172.21.15.47:0/581019479 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 10 monclient(hunting): get_auth_request con 0x7f4c700b43c0 auth_method 0 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4c7013e890 auth_method 0 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4c70144b80 auth_method 0 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.151+0000 7f4c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.152+0000 7f4c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 10 monclient(hunting): handle_auth_done global_id 22372 payload 824 2024-03-30T07:44:56.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 -- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4c70144b80 msgr2=0x7f4c701d9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4c70144b80 0x7f4c701d9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 -- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c7013e890 msgr2=0x7f4c70142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4c7013e890 0x7f4c70142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 monclient: found mon.noname-b 2024-03-30T07:44:56.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.154+0000 7f4c6effd640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680094d0 con 0x7f4c700b43c0 2024-03-30T07:44:56.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.157+0000 7f4c6effd640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c701377c0 secure :-1 s=READY pgs=5519 cs=0 l=1 rev1=1 crypto rx=0x7f4c6802b950 tx=0x7f4c680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.157+0000 7f4c677fe640 10 client.?.objecter ms_handle_connect 0x7f4c700b43c0 2024-03-30T07:44:56.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.157+0000 7f4c677fe640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6803abe0 con 0x7f4c700b43c0 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: dump: 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.170 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6803ad60 con 0x7f4c700b43c0 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 5 monclient: authenticate success, global_id 22372 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 10 monclient: _renew_subs 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c68044630 con 0x7f4c700b43c0 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c70140dd0 con 0x7f4c700b43c0 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:56.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c677fe640 10 monclient: dump: 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.172 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c6cff9640 4 set_mon_vals no callback set 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 10 client.22372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.160+0000 7f4c75699640 10 monclient: _renew_subs 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c701da4c0 con 0x7f4c700b43c0 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 1 librados: init done 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 10 client.22372.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 15 client.22372.objecter _session_command_op_assign -1 1 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 20 client.22372.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 10 client.22372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.161+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c7014bf10 con 0x7f4c700b43c0 2024-03-30T07:44:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.181+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4c6804d450 con 0x7f4c700b43c0 2024-03-30T07:44:56.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.182+0000 7f4c677fe640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4c5006c610 0x7f4c5006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.182+0000 7f4c6e7fc640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4c5006c610 0x7f4c5006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.182+0000 7f4c6e7fc640 10 monclient: get_auth_request con 0x7f4c5006c610 auth_method 0 2024-03-30T07:44:56.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.183+0000 7f4c677fe640 10 client.22372.objecter ms_handle_connect 0x7f4c5006c610 2024-03-30T07:44:56.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.183+0000 7f4c6e7fc640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4c5006c610 0x7f4c5006ead0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c7013cb20 tx=0x7f4c58009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f4c6802d040 con 0x7f4c700b43c0 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 10 client.22372.objecter ms_dispatch 0x7f4c70000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 3 client.22372.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 3 client.22372.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 10 client.22372.objecter checking command 1 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f4c50073fe0 0x7f4c50076480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter _get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 15 client.22372.objecter _session_command_op_remove -1 1 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter _get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 15 client.22372.objecter _session_command_op_assign 3 1 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 4 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 10 client.22372.objecter _send_command 1 2024-03-30T07:44:56.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c50076b50 con 0x7f4c50073fe0 2024-03-30T07:44:56.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 20 client.22372.objecter dump_active .. 0 homeless 2024-03-30T07:44:56.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c68077060 con 0x7f4c700b43c0 2024-03-30T07:44:56.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c677fe640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:56.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.184+0000 7f4c6f7fe640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f4c50073fe0 0x7f4c50076480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.185+0000 7f4c6f7fe640 10 monclient: get_auth_request con 0x7f4c50073fe0 auth_method 0 2024-03-30T07:44:56.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.187+0000 7f4c6f7fe640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f4c50073fe0 0x7f4c50076480 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.187+0000 7f4c677fe640 10 client.22372.objecter ms_handle_connect 0x7f4c50073fe0 2024-03-30T07:44:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.211+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== osd.3 v2:172.21.15.47:6810/35206 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4c50076b50 con 0x7f4c50073fe0 2024-03-30T07:44:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.211+0000 7f4c677fe640 10 client.22372.objecter ms_dispatch 0x7f4c70000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.211+0000 7f4c677fe640 10 client.22372.objecter _finish_command 1 = system:0 2024-03-30T07:44:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.211+0000 7f4c677fe640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 4 2024-03-30T07:44:56.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.211+0000 7f4c677fe640 15 client.22372.objecter _session_command_op_remove 3 1 2024-03-30T07:44:56.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 10 client.22372.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 15 client.22372.objecter _session_command_op_assign -1 2 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 20 client.22372.objecter _get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 20 client.22372.objecter _get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 15 client.22372.objecter _session_command_op_remove -1 2 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 20 client.22372.objecter get_session s=0x7f4c50073910 osd=3 3 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 15 client.22372.objecter _session_command_op_assign 3 2 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 4 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 10 client.22372.objecter _send_command 2 2024-03-30T07:44:56.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.224+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 --> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4c70144970 con 0x7f4c50073fe0 2024-03-30T07:44:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c677fe640 1 -- 172.21.15.47:0/581019479 <== osd.3 v2:172.21.15.47:6810/35206 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4c70144970 con 0x7f4c50073fe0 2024-03-30T07:44:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c677fe640 10 client.22372.objecter ms_dispatch 0x7f4c70000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c677fe640 10 client.22372.objecter _finish_command 2 = system:0 2024-03-30T07:44:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c677fe640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 4 2024-03-30T07:44:56.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c677fe640 15 client.22372.objecter _session_command_op_remove 3 2 2024-03-30T07:44:56.243 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:56.243 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:56.243 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 10 librados: watch_flush enter 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 10 librados: watch_flush exit 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 10 client.22372.objecter close_session for osd.3 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f4c50073fe0 msgr2=0x7f4c50076480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.47:6810/35206,v1:172.21.15.47:6811/35206] conn(0x7f4c50073fe0 0x7f4c50076480 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 20 client.22372.objecter put_session s=0x7f4c50073910 osd=3 1 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 20 client.22372.objecter shutdown clearing up homeless session... 2024-03-30T07:44:56.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.240+0000 7f4c75699640 10 client.22372.objecter successfully canceled tick 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4c5006c610 msgr2=0x7f4c5006ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4c5006c610 0x7f4c5006ead0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4c7013cb20 tx=0x7f4c58009190 comp rx=0 tx=0).stop 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 10 monclient: shutdown 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 msgr2=0x7f4c701377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 --2- 172.21.15.47:0/581019479 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4c700b43c0 0x7f4c701377c0 secure :-1 s=READY pgs=5519 cs=0 l=1 rev1=1 crypto rx=0x7f4c6802b950 tx=0x7f4c680046c0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c6f7fe640 1 -- 172.21.15.47:0/581019479 reap_dead start 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 shutdown_connections 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 >> 172.21.15.47:0/581019479 conn(0x7f4c70071930 msgr2=0x7f4c7013b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 shutdown_connections 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 -- 172.21.15.47:0/581019479 wait complete. 2024-03-30T07:44:56.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.241+0000 7f4c75699640 1 librados: shutdown 2024-03-30T07:44:56.254 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-30T07:44:56.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.353+0000 7f128b705640 10 monclient: get_monmap_and_config 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.353+0000 7f128b705640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.353+0000 7f128b705640 10 monclient: monmap: 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:56.354658+0000 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:56.354658+0000 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:56.356 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:56.357 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:56.357 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.354+0000 7f128b705640 1 Processor -- start 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.354+0000 7f128b705640 1 -- start start 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.354+0000 7f128b705640 10 monclient: init 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1284071be0 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): picked mon.noname-b con 0x7f12840b0ca0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f128413e820 0x7f128413ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): picked mon.noname-c con 0x7f128413e820 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12840a8cc0 con 0x7f1284071be0 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1284071fc0 con 0x7f128413e820 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12840aa380 con 0x7f12840b0ca0 2024-03-30T07:44:56.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f128b705640 10 monclient(hunting): authenticate will time out at 3222.470507s 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.355+0000 7f1288ca0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60094/0 (socket says 172.21.15.47:60094) 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 1 -- 172.21.15.47:0/2744353165 learned_addr learned my addr 172.21.15.47:0/2744353165 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): get_auth_request con 0x7f12840b0ca0 auth_method 0 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1289ca2640 1 --2- 172.21.15.47:0/2744353165 >> v2:172.21.15.52:3300/0 conn(0x7f128413e820 0x7f128413ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.356+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.357+0000 7f1289ca2640 10 monclient(hunting): get_auth_request con 0x7f128413e820 auth_method 0 2024-03-30T07:44:56.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.357+0000 7f1289ca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.357+0000 7f1289ca2640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.357+0000 7f1289ca2640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 10 monclient(hunting): handle_auth_done global_id 17099 payload 274 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 1 -- 172.21.15.47:0/2744353165 >> v2:172.21.15.52:3300/0 conn(0x7f128413e820 msgr2=0x7f128413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 1 --2- 172.21.15.47:0/2744353165 >> v2:172.21.15.52:3300/0 conn(0x7f128413e820 0x7f128413ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 1 -- 172.21.15.47:0/2744353165 >> v1:172.21.15.52:6789/0 conn(0x7f1284071be0 legacy=0x7f1284144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 1 monclient: found mon.noname-b 2024-03-30T07:44:56.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.358+0000 7f1288ca0640 1 -- 172.21.15.47:0/2744353165 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840a8970 con 0x7f12840b0ca0 2024-03-30T07:44:56.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.360+0000 7f1288ca0640 1 --2- 172.21.15.47:0/2744353165 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f1278009870 tx=0x7f127802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b866935d4b123bde server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.363+0000 7f1283fff640 1 -- 172.21.15.47:0/2744353165 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278030840 con 0x7f12840b0ca0 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: dump: 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.366 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 1 -- 172.21.15.47:0/2744353165 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12780309a0 con 0x7f12840b0ca0 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 1 -- 172.21.15.47:0/2744353165 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278030d30 con 0x7f12840b0ca0 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 5 monclient: authenticate success, global_id 17099 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 4 set_mon_vals no callback set 2024-03-30T07:44:56.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f1283fff640 10 monclient: dump: 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 10 monclient: shutdown 2024-03-30T07:44:56.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 -- 172.21.15.47:0/2744353165 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 msgr2=0x7f12840a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 --2- 172.21.15.47:0/2744353165 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f1278009870 tx=0x7f127802b980 comp rx=0 tx=0).stop 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 -- 172.21.15.47:0/2744353165 shutdown_connections 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 --2- 172.21.15.47:0/2744353165 >> v2:172.21.15.52:3300/0 conn(0x7f128413e820 0x7f128413ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 --2- 172.21.15.47:0/2744353165 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12840b0ca0 0x7f12840a9150 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 -- 172.21.15.47:0/2744353165 >> 172.21.15.47:0/2744353165 conn(0x7f12840b73d0 msgr2=0x7f12840b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 -- 172.21.15.47:0/2744353165 shutdown_connections 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.364+0000 7f128b705640 1 -- 172.21.15.47:0/2744353165 wait complete. 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 10 monclient: monmap: 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:56.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 librados: starting msgr at 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 librados: starting objecter 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 Processor -- start 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 -- start start 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 librados: setting wanted keys 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 1 librados: calling monclient init 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.365+0000 7f128b705640 10 monclient: init 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:44:56.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1284071be0 0x7f128413f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): picked mon.noname-a con 0x7f1284071be0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 0x7f1284140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): picked mon.noname-b con 0x7f12840b0ca0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): picked mon.noname-c con 0x7f128413e820 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12841408d0 con 0x7f1284071be0 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12841d6610 con 0x7f12840b0ca0 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1284137bf0 con 0x7f128413e820 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f128b705640 10 monclient(hunting): authenticate will time out at 3222.481546s 2024-03-30T07:44:56.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.366+0000 7f1289ca2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 0x7f1284140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1284071be0 0x7f128413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60108/0 (socket says 172.21.15.47:60108) 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 1 -- 172.21.15.47:0/311654949 learned_addr learned my addr 172.21.15.47:0/311654949 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): get_auth_request con 0x7f128413e820 auth_method 0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): get_auth_request con 0x7f12840b0ca0 auth_method 0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): get_auth_request con 0x7f1284071be0 auth_method 0 2024-03-30T07:44:56.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1289ca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f1288ca0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.367+0000 7f12894a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 10 monclient(hunting): handle_auth_done global_id 22381 payload 824 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 -- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 msgr2=0x7f12841d6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 -- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1284071be0 msgr2=0x7f128413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1284071be0 0x7f128413f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 monclient: found mon.noname-b 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1288ca0640 1 -- 172.21.15.47:0/311654949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780094d0 con 0x7f12840b0ca0 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.370+0000 7f1289ca2640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.372+0000 7f1288ca0640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 0x7f1284140fd0 secure :-1 s=READY pgs=5521 cs=0 l=1 rev1=1 crypto rx=0x7f1278033000 tx=0x7f1278004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.372+0000 7f1281ffb640 10 client.?.objecter ms_handle_connect 0x7f12840b0ca0 2024-03-30T07:44:56.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.372+0000 7f1281ffb640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278047070 con 0x7f12840b0ca0 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: dump: 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.379 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1278039b50 con 0x7f12840b0ca0 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1278043400 con 0x7f12840b0ca0 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f1281ffb640 10 monclient: dump: 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 5 monclient: authenticate success, global_id 22381 2024-03-30T07:44:56.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 10 monclient: _renew_subs 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1284137370 con 0x7f12840b0ca0 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f12837fe640 4 set_mon_vals no callback set 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 10 client.22381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 10 monclient: _renew_subs 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12841380b0 con 0x7f12840b0ca0 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.376+0000 7f128b705640 1 librados: init done 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.377+0000 7f128b705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.377+0000 7f128b705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.377+0000 7f128b705640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.377+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1284137500 con 0x7f12840b0ca0 2024-03-30T07:44:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.641+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1278043560 con 0x7f12840b0ca0 2024-03-30T07:44:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.642+0000 7f1281ffb640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 0x7f127006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.642+0000 7f12894a1640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 0x7f127006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.642+0000 7f12894a1640 10 monclient: get_auth_request con 0x7f127006c5e0 auth_method 0 2024-03-30T07:44:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.642+0000 7f12894a1640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 0x7f127006eaa0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f127c0012e0 tx=0x7f127c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.642+0000 7f1281ffb640 10 client.22381.objecter ms_handle_connect 0x7f127006c5e0 2024-03-30T07:44:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.644+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f127803e070 con 0x7f12840b0ca0 2024-03-30T07:44:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.644+0000 7f1281ffb640 10 client.22381.objecter ms_dispatch 0x7f1284000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.644+0000 7f1281ffb640 3 client.22381.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.644+0000 7f1281ffb640 3 client.22381.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:56.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.644+0000 7f1281ffb640 20 client.22381.objecter dump_active .. 0 homeless 2024-03-30T07:44:56.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.655+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f1278077950 con 0x7f12840b0ca0 2024-03-30T07:44:56.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.655+0000 7f1281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:56.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.655+0000 7f1281ffb640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:56.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.836+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f12840b37b0 con 0x7f127006c5e0 2024-03-30T07:44:56.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.836+0000 7f1281ffb640 1 -- 172.21.15.47:0/311654949 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f127c00fe00 con 0x7f127006c5e0 2024-03-30T07:44:56.838 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 3 to deep-scrub 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 10 librados: watch_flush enter 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 10 librados: watch_flush exit 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 20 client.22381.objecter shutdown clearing up homeless session... 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 10 client.22381.objecter successfully canceled tick 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 msgr2=0x7f127006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 0x7f127006eaa0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f127c0012e0 tx=0x7f127c009040 comp rx=0 tx=0).stop 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 10 monclient: shutdown 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 msgr2=0x7f1284140fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 0x7f1284140fd0 secure :-1 s=READY pgs=5521 cs=0 l=1 rev1=1 crypto rx=0x7f1278033000 tx=0x7f1278004730 comp rx=0 tx=0).stop 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 shutdown_connections 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f127006c5e0 0x7f127006eaa0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f128413e820 0x7f12841d6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f12840b0ca0 0x7f1284140fd0 unknown :-1 s=CLOSED pgs=5521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 --2- 172.21.15.47:0/311654949 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1284071be0 0x7f128413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 >> 172.21.15.47:0/311654949 conn(0x7f12840b73d0 msgr2=0x7f128413d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 shutdown_connections 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 -- 172.21.15.47:0/311654949 wait complete. 2024-03-30T07:44:56.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.838+0000 7f128b705640 1 librados: shutdown 2024-03-30T07:44:56.853 INFO:tasks.ceph:Scrubbing osd.4 2024-03-30T07:44:56.853 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:56.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.959+0000 7f6ba894e640 10 monclient: get_monmap_and_config 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.959+0000 7f6ba894e640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.959+0000 7f6ba894e640 10 monclient: monmap: 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:56.960515+0000 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:56.960515+0000 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:56.962 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.960+0000 7f6ba894e640 1 Processor -- start 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.960+0000 7f6ba894e640 1 -- start start 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.960+0000 7f6ba894e640 10 monclient: init 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-03-30T07:44:56.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6ba40b43c0 0x7f6ba40a8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba40b43c0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6ba413e890 0x7f6ba413ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba413e890 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba4144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba4144b80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba40a8da0 con 0x7f6ba413e890 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba40b47a0 con 0x7f6ba40b43c0 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba413f210 con 0x7f6ba4144b80 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba894e640 10 monclient(hunting): authenticate will time out at 3223.076360s 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.961+0000 7f6ba259c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6ba40b43c0 0x7f6ba40a8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f6ba40b43c0 0x7f6ba40a8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:52128/0 (socket says 172.21.15.47:52128) 2024-03-30T07:44:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 1 -- 172.21.15.47:0/1548390378 learned_addr learned my addr 172.21.15.47:0/1548390378 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): get_auth_request con 0x7f6ba40b43c0 auth_method 0 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.962+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6b9bfff640 1 --2- 172.21.15.47:0/1548390378 >> v2:172.21.15.47:3300/0 conn(0x7f6ba413e890 0x7f6ba413ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 10 monclient(hunting): handle_auth_done global_id 22384 payload 274 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 -- 172.21.15.47:0/1548390378 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 msgr2=0x7f6ba4144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 --2- 172.21.15.47:0/1548390378 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba4144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 -- 172.21.15.47:0/1548390378 >> v2:172.21.15.47:3300/0 conn(0x7f6ba413e890 msgr2=0x7f6ba413ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 --2- 172.21.15.47:0/1548390378 >> v2:172.21.15.47:3300/0 conn(0x7f6ba413e890 0x7f6ba413ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 monclient: found mon.noname-c 2024-03-30T07:44:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.964+0000 7f6ba259c640 1 -- 172.21.15.47:0/1548390378 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba4145500 con 0x7f6ba40b43c0 2024-03-30T07:44:56.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.966+0000 7f6ba259c640 1 --2- 172.21.15.47:0/1548390378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba40a8860 secure :-1 s=READY pgs=5522 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40b81c0 tx=0x7f6b9402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb8b64b12ae15e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 1 -- 172.21.15.47:0/1548390378 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940317e0 con 0x7f6ba40b43c0 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: dump: 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.977 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 1 -- 172.21.15.47:0/1548390378 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94031940 con 0x7f6ba40b43c0 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 1 -- 172.21.15.47:0/1548390378 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94031cd0 con 0x7f6ba40b43c0 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba894e640 5 monclient: authenticate success, global_id 22384 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba894e640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:56.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba894e640 4 set_mon_vals no callback set 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.970+0000 7f6ba1d9b640 10 monclient: dump: 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 10 monclient: shutdown 2024-03-30T07:44:56.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 -- 172.21.15.47:0/1548390378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 msgr2=0x7f6ba40a8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 --2- 172.21.15.47:0/1548390378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba40a8860 secure :-1 s=READY pgs=5522 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40b81c0 tx=0x7f6b9402b880 comp rx=0 tx=0).stop 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 -- 172.21.15.47:0/1548390378 shutdown_connections 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 --2- 172.21.15.47:0/1548390378 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba4144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 --2- 172.21.15.47:0/1548390378 >> v2:172.21.15.47:3300/0 conn(0x7f6ba413e890 0x7f6ba413ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 --2- 172.21.15.47:0/1548390378 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba40a8860 unknown :-1 s=CLOSED pgs=5522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 -- 172.21.15.47:0/1548390378 >> 172.21.15.47:0/1548390378 conn(0x7f6ba4071930 msgr2=0x7f6ba4071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 -- 172.21.15.47:0/1548390378 shutdown_connections 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.971+0000 7f6ba894e640 1 -- 172.21.15.47:0/1548390378 wait complete. 2024-03-30T07:44:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 10 monclient: build_initial_monmap 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 10 monclient: monmap: 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 librados: starting msgr at 2024-03-30T07:44:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 librados: starting objecter 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 Processor -- start 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 -- start start 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 librados: setting wanted keys 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 1 librados: calling monclient init 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.972+0000 7f6ba894e640 10 monclient: init 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba4139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba40b43c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ba413e890 0x7f6ba40b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba413e890 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba40b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba4144b80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba413f790 con 0x7f6ba413e890 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40b7010 con 0x7f6ba40b43c0 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4141ea0 con 0x7f6ba4144b80 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.973+0000 7f6ba894e640 10 monclient(hunting): authenticate will time out at 3223.088468s 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba4139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba4139690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:52134/0 (socket says 172.21.15.47:52134) 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 1 -- 172.21.15.47:0/254046161 learned_addr learned my addr 172.21.15.47:0/254046161 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 10 monclient(hunting): get_auth_request con 0x7f6ba40b43c0 auth_method 0 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.974+0000 7f6ba259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6ba259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6b9bfff640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ba413e890 0x7f6ba40b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6b9bfff640 10 monclient(hunting): get_auth_request con 0x7f6ba413e890 auth_method 0 2024-03-30T07:44:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.975+0000 7f6b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.976+0000 7f6ba2d9d640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba40b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.976+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.976+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.976+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba2d9d640 10 monclient(hunting): get_auth_request con 0x7f6ba4144b80 auth_method 0 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba2d9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 10 monclient(hunting): handle_auth_done global_id 22387 payload 824 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 -- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 msgr2=0x7f6ba40b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6ba4144b80 0x7f6ba40b6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 -- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ba413e890 msgr2=0x7f6ba40b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6ba413e890 0x7f6ba40b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 10 monclient: _finish_hunting 0 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 monclient: found mon.noname-b 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.977+0000 7f6ba259c640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b940094d0 con 0x7f6ba40b43c0 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.980+0000 7f6ba259c640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba4139690 secure :-1 s=READY pgs=5523 cs=0 l=1 rev1=1 crypto rx=0x7f6b94000c00 tx=0x7f6b940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.980+0000 7f6b9affd640 10 client.?.objecter ms_handle_connect 0x7f6ba40b43c0 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.980+0000 7f6b9affd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94032980 con 0x7f6ba40b43c0 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: dump: 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: _finish_auth 0 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b94032ae0 con 0x7f6ba40b43c0 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6ba894e640 5 monclient: authenticate success, global_id 22387 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6ba894e640 10 monclient: _renew_subs 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6ba894e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba4142060 con 0x7f6ba40b43c0 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94032e90 con 0x7f6ba40b43c0 2024-03-30T07:44:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6b9affd640 10 monclient: dump: 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.983+0000 7f6ba159a640 4 set_mon_vals no callback set 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 client.22387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 monclient: _renew_subs 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba4142540 con 0x7f6ba40b43c0 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 1 librados: init done 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 client.22387.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 15 client.22387.objecter _session_command_op_assign -1 1 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 20 client.22387.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 client.22387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:56.984+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ba40a8f80 con 0x7f6ba40b43c0 2024-03-30T07:44:57.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.003+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6b9404e450 con 0x7f6ba40b43c0 2024-03-30T07:44:57.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.004+0000 7f6b9affd640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6b8406c5e0 0x7f6b8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.004+0000 7f6b9bfff640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6b8406c5e0 0x7f6b8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.004+0000 7f6b9bfff640 10 monclient: get_auth_request con 0x7f6b8406c5e0 auth_method 0 2024-03-30T07:44:57.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.004+0000 7f6b9affd640 10 client.22387.objecter ms_handle_connect 0x7f6b8406c5e0 2024-03-30T07:44:57.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.004+0000 7f6b9bfff640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6b8406c5e0 0x7f6b8406eaa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6ba41d4490 tx=0x7f6b8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f6b9407b5a0 con 0x7f6ba40b43c0 2024-03-30T07:44:57.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 10 client.22387.objecter ms_dispatch 0x7f6ba4000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:57.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 3 client.22387.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:57.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 3 client.22387.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:57.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 10 client.22387.objecter checking command 1 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7f6b84073e40 0x7f6b840762e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter _get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 15 client.22387.objecter _session_command_op_remove -1 1 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter _get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 15 client.22387.objecter _session_command_op_assign 4 1 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 4 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 10 client.22387.objecter _send_command 1 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b840769b0 con 0x7f6b84073e40 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6b9affd640 20 client.22387.objecter dump_active .. 0 homeless 2024-03-30T07:44:57.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.007+0000 7f6ba2d9d640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7f6b84073e40 0x7f6b840762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.008+0000 7f6ba2d9d640 10 monclient: get_auth_request con 0x7f6b84073e40 auth_method 0 2024-03-30T07:44:57.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.010+0000 7f6ba2d9d640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7f6b84073e40 0x7f6b840762e0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.010+0000 7f6b9affd640 10 client.22387.objecter ms_handle_connect 0x7f6b84073e40 2024-03-30T07:44:57.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.051+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== osd.4 v2:172.21.15.52:6818/35313 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6b840769b0 con 0x7f6b84073e40 2024-03-30T07:44:57.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.051+0000 7f6b9affd640 10 client.22387.objecter ms_dispatch 0x7f6ba4000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:57.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.051+0000 7f6b9affd640 10 client.22387.objecter _finish_command 1 = system:0 2024-03-30T07:44:57.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.051+0000 7f6b9affd640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 4 2024-03-30T07:44:57.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.051+0000 7f6b9affd640 15 client.22387.objecter _session_command_op_remove 4 1 2024-03-30T07:44:57.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 10 client.22387.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:57.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 15 client.22387.objecter _session_command_op_assign -1 2 2024-03-30T07:44:57.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 20 client.22387.objecter _get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 20 client.22387.objecter _get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 15 client.22387.objecter _session_command_op_remove -1 2 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 20 client.22387.objecter get_session s=0x7f6b94032e90 osd=4 3 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 15 client.22387.objecter _session_command_op_assign 4 2 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 4 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 10 client.22387.objecter _send_command 2 2024-03-30T07:44:57.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.063+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 --> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6ba4067480 con 0x7f6b84073e40 2024-03-30T07:44:57.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== osd.4 v2:172.21.15.52:6818/35313 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6ba4067480 con 0x7f6b84073e40 2024-03-30T07:44:57.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 10 client.22387.objecter ms_dispatch 0x7f6ba4000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 10 client.22387.objecter _finish_command 2 = system:0 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 4 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 15 client.22387.objecter _session_command_op_remove 4 2 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 1 -- 172.21.15.47:0/254046161 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b940771e0 con 0x7f6ba40b43c0 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6b9affd640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:57.089 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.086+0000 7f6ba894e640 10 librados: watch_flush enter 2024-03-30T07:44:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 10 librados: watch_flush exit 2024-03-30T07:44:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 10 client.22387.objecter close_session for osd.4 2024-03-30T07:44:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7f6b84073e40 msgr2=0x7f6b840762e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6818/35313,v1:172.21.15.52:6819/35313] conn(0x7f6b84073e40 0x7f6b840762e0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 20 client.22387.objecter put_session s=0x7f6b94032e90 osd=4 1 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 20 client.22387.objecter shutdown clearing up homeless session... 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 10 client.22387.objecter successfully canceled tick 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6b8406c5e0 msgr2=0x7f6b8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f6b8406c5e0 0x7f6b8406eaa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6ba41d4490 tx=0x7f6b8c008040 comp rx=0 tx=0).stop 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 10 monclient: shutdown 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 msgr2=0x7f6ba4139690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 --2- 172.21.15.47:0/254046161 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f6ba40b43c0 0x7f6ba4139690 secure :-1 s=READY pgs=5523 cs=0 l=1 rev1=1 crypto rx=0x7f6b94000c00 tx=0x7f6b940046c0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba2d9d640 1 -- 172.21.15.47:0/254046161 reap_dead start 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 shutdown_connections 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 >> 172.21.15.47:0/254046161 conn(0x7f6ba4071930 msgr2=0x7f6ba413ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 shutdown_connections 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 -- 172.21.15.47:0/254046161 wait complete. 2024-03-30T07:44:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.087+0000 7f6ba894e640 1 librados: shutdown 2024-03-30T07:44:57.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.202+0000 7fbab0c99640 10 monclient: get_monmap_and_config 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.202+0000 7fbab0c99640 10 monclient: build_initial_monmap 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.202+0000 7fbab0c99640 10 monclient: monmap: 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:57.203160+0000 2024-03-30T07:44:57.204 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:57.203160+0000 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:57.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.203+0000 7fbab0c99640 1 Processor -- start 2024-03-30T07:44:57.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.203+0000 7fbab0c99640 1 -- start start 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.203+0000 7fbab0c99640 10 monclient: init 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaac0b4280 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 0x7fbaac13ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-b con 0x7fbaac13e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaac141670 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaac13f2b0 con 0x7fbaac0b4280 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaac0aa3c0 con 0x7fbaac141670 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac0b4660 con 0x7fbaac13e930 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.204+0000 7fbab0c99640 10 monclient(hunting): authenticate will time out at 3223.319619s 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaa9d9b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 0x7fbaac13ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaaad9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fbaac141670 0x7fbaac141ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.47:44828/0 (socket says 172.21.15.47:44828) 2024-03-30T07:44:57.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaaad9d640 1 -- 172.21.15.47:0/1299184748 learned_addr learned my addr 172.21.15.47:0/1299184748 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:57.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaa9d9b640 10 monclient(hunting): get_auth_request con 0x7fbaac13e930 auth_method 0 2024-03-30T07:44:57.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaa9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaa9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.205+0000 7fbaa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.206+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.206+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.206+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.207+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_done global_id 17108 payload 274 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1299184748 >> v1:172.21.15.52:6789/0 conn(0x7fbaac141670 legacy=0x7fbaac141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1299184748 >> v1:172.21.15.47:6789/0 conn(0x7fbaac0b4280 legacy=0x7fbaac0a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 1 monclient: found mon.noname-b 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:57.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.208+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1299184748 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac0aa630 con 0x7fbaac13e930 2024-03-30T07:44:57.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.209+0000 7fbaa9d9b640 1 --2- 172.21.15.47:0/1299184748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 0x7fbaac13ed70 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00097d0 tx=0x7fbaa002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d691fbf2bd1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 1 -- 172.21.15.47:0/1299184748 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0031800 con 0x7fbaac13e930 2024-03-30T07:44:57.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: dump: 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: _finish_auth 0 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: _check_auth_tickets 2024-03-30T07:44:57.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 1 -- 172.21.15.47:0/1299184748 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa0031960 con 0x7fbaac13e930 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbab0c99640 5 monclient: authenticate success, global_id 17108 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbab0c99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbab0c99640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 4 set_mon_vals no callback set 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.213+0000 7fbaa959a640 1 -- 172.21.15.47:0/1299184748 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0031cf0 con 0x7fbaac13e930 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbaa959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbaa959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbaa959a640 10 monclient: dump: 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.217 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.218 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 10 monclient: shutdown 2024-03-30T07:44:57.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 -- 172.21.15.47:0/1299184748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 msgr2=0x7fbaac13ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1299184748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 0x7fbaac13ed70 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00097d0 tx=0x7fbaa002b8a0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 -- 172.21.15.47:0/1299184748 shutdown_connections 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1299184748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac13e930 0x7fbaac13ed70 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 -- 172.21.15.47:0/1299184748 >> 172.21.15.47:0/1299184748 conn(0x7fbaac065e00 msgr2=0x7fbaac066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 -- 172.21.15.47:0/1299184748 shutdown_connections 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.214+0000 7fbab0c99640 1 -- 172.21.15.47:0/1299184748 wait complete. 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.215+0000 7fbab0c99640 10 monclient: build_initial_monmap 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.215+0000 7fbab0c99640 10 monclient: monmap: 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:57.222 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.215+0000 7fbab0c99640 1 librados: starting msgr at 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.215+0000 7fbab0c99640 1 librados: starting objecter 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 Processor -- start 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 -- start start 2024-03-30T07:44:57.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 librados: setting wanted keys 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 librados: calling monclient init 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient: init 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaac0b4280 0x7fbaac134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-a con 0x7fbaac0b4280 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 0x7fbaac140c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-b con 0x7fbaac13e930 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 0x7fbaac142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): picked mon.noname-c con 0x7fbaac141670 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac134380 con 0x7fbaac0b4280 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac1358f0 con 0x7fbaac13e930 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaac135bd0 con 0x7fbaac141670 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.216+0000 7fbab0c99640 10 monclient(hunting): authenticate will time out at 3223.331791s 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 0x7fbaac142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaa9d9b640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 0x7fbaac140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaa59c640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaac0b4280 0x7fbaac134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 0x7fbaac142e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:52636/0 (socket says 172.21.15.47:52636) 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 1 -- 172.21.15.47:0/1271744988 learned_addr learned my addr 172.21.15.47:0/1271744988 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 10 monclient(hunting): get_auth_request con 0x7fbaac141670 auth_method 0 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaa59c640 10 monclient(hunting): get_auth_request con 0x7fbaac0b4280 auth_method 0 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaa59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaa9d9b640 10 monclient(hunting): get_auth_request con 0x7fbaac13e930 auth_method 0 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaa9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaa9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.217+0000 7fbaa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.218+0000 7fbaaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 10 monclient(hunting): handle_auth_done global_id 22393 payload 824 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 msgr2=0x7fbaac142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 0x7fbaac142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaac0b4280 msgr2=0x7fbaac134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaac0b4280 0x7fbaac134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 10 monclient: _finish_hunting 0 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.220+0000 7fbaa9d9b640 1 monclient: found mon.noname-b 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.221+0000 7fbaa9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.221+0000 7fbaa9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.221+0000 7fbaa9d9b640 1 -- 172.21.15.47:0/1271744988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0009400 con 0x7fbaac13e930 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.223+0000 7fbaa9d9b640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 0x7fbaac140c90 secure :-1 s=READY pgs=5525 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0033000 tx=0x7fbaa00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.223+0000 7fba9affd640 10 client.?.objecter ms_handle_connect 0x7fbaac13e930 2024-03-30T07:44:57.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.223+0000 7fba9affd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:57.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa003bd80 con 0x7fbaac13e930 2024-03-30T07:44:57.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:57.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: dump: 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: _finish_auth 0 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:57.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa00453f0 con 0x7fbaac13e930 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00457a0 con 0x7fbaac13e930 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fba9affd640 10 monclient: dump: 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.231 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 5 monclient: authenticate success, global_id 22393 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 10 monclient: _renew_subs 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaac1350a0 con 0x7fbaac13e930 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbaa8d99640 4 set_mon_vals no callback set 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 10 client.22393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 10 monclient: _renew_subs 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaac1433c0 con 0x7fbaac13e930 2024-03-30T07:44:57.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.227+0000 7fbab0c99640 1 librados: init done 2024-03-30T07:44:57.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.228+0000 7fbab0c99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:57.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.228+0000 7fbab0c99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:57.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.228+0000 7fbab0c99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.228+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaac04e030 con 0x7fbaac13e930 2024-03-30T07:44:57.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.493+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbaa004e450 con 0x7fbaac13e930 2024-03-30T07:44:57.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.494+0000 7fba9affd640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 0x7fba8c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.494+0000 7fbaaa59c640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 0x7fba8c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.494+0000 7fbaaa59c640 10 monclient: get_auth_request con 0x7fba8c06c590 auth_method 0 2024-03-30T07:44:57.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.494+0000 7fbaaa59c640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 0x7fba8c06ea50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fba9c000f60 tx=0x7fba9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.494+0000 7fba9affd640 10 client.22393.objecter ms_handle_connect 0x7fba8c06c590 2024-03-30T07:44:57.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.496+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fbaa002e070 con 0x7fbaac13e930 2024-03-30T07:44:57.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.496+0000 7fba9affd640 10 client.22393.objecter ms_dispatch 0x7fbaac000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:57.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.496+0000 7fba9affd640 3 client.22393.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:57.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.496+0000 7fba9affd640 3 client.22393.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:57.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.496+0000 7fba9affd640 20 client.22393.objecter dump_active .. 0 homeless 2024-03-30T07:44:57.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.507+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fbaa0076ec0 con 0x7fbaac13e930 2024-03-30T07:44:57.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.507+0000 7fba9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:57.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.507+0000 7fba9affd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:57.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.689+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaac0b51c0 con 0x7fba8c06c590 2024-03-30T07:44:57.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.689+0000 7fba9affd640 1 -- 172.21.15.47:0/1271744988 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbaac0b51c0 con 0x7fba8c06c590 2024-03-30T07:44:57.691 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 4 to deep-scrub 2024-03-30T07:44:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 10 librados: watch_flush enter 2024-03-30T07:44:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 10 librados: watch_flush exit 2024-03-30T07:44:57.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 20 client.22393.objecter shutdown clearing up homeless session... 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 10 client.22393.objecter successfully canceled tick 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 msgr2=0x7fba8c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 0x7fba8c06ea50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fba9c000f60 tx=0x7fba9c009040 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.691+0000 7fbab0c99640 10 monclient: shutdown 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 msgr2=0x7fbaac140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 0x7fbaac140c90 secure :-1 s=READY pgs=5525 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0033000 tx=0x7fbaa00046c0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 shutdown_connections 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fba8c06c590 0x7fba8c06ea50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbaac141670 0x7fbaac142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fbaac13e930 0x7fbaac140c90 unknown :-1 s=CLOSED pgs=5525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 --2- 172.21.15.47:0/1271744988 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbaac0b4280 0x7fbaac134b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 >> 172.21.15.47:0/1271744988 conn(0x7fbaac065e00 msgr2=0x7fbaac0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 shutdown_connections 2024-03-30T07:44:57.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 -- 172.21.15.47:0/1271744988 wait complete. 2024-03-30T07:44:57.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.692+0000 7fbab0c99640 1 librados: shutdown 2024-03-30T07:44:57.704 INFO:tasks.ceph:Scrubbing osd.5 2024-03-30T07:44:57.705 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:57.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 10 monclient: get_monmap_and_config 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 10 monclient: build_initial_monmap 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 10 monclient: monmap: 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:57.809767+0000 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:57.809767+0000 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:57.811 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:57.812 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:57.812 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:57.812 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 1 Processor -- start 2024-03-30T07:44:57.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 1 -- start start 2024-03-30T07:44:57.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.809+0000 7fd8aa3c4640 10 monclient: init 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd8a40b43c0 0x7fd8a40a8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd8a40b43c0 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8a413e890 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a4144b80 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a4145550 con 0x7fd8a413e890 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a40a8da0 con 0x7fd8a40b43c0 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a40b47a0 con 0x7fd8a4144b80 2024-03-30T07:44:57.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.810+0000 7fd8aa3c4640 10 monclient(hunting): authenticate will time out at 3223.925604s 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a3fff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd8a40b43c0 0x7fd8a40a8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:52650/0 (socket says 172.21.15.47:52650) 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 1 -- 172.21.15.47:0/3370361214 learned_addr learned my addr 172.21.15.47:0/3370361214 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): get_auth_request con 0x7fd8a4144b80 auth_method 0 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a3fff640 10 monclient(hunting): get_auth_request con 0x7fd8a40b43c0 auth_method 0 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.811+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.812+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.812+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.812+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.812+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 10 monclient(hunting): handle_auth_done global_id 17114 payload 274 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 1 -- 172.21.15.47:0/3370361214 >> v2:172.21.15.47:3300/0 conn(0x7fd8a40b43c0 msgr2=0x7fd8a40a8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 1 --2- 172.21.15.47:0/3370361214 >> v2:172.21.15.47:3300/0 conn(0x7fd8a40b43c0 0x7fd8a40a8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 1 -- 172.21.15.47:0/3370361214 >> v1:172.21.15.47:6789/0 conn(0x7fd8a413e890 legacy=0x7fd8a413ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 10 monclient: _finish_hunting 0 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 1 monclient: found mon.noname-b 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:57.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.813+0000 7fd8a8961640 1 -- 172.21.15.47:0/3370361214 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a40aa3f0 con 0x7fd8a4144b80 2024-03-30T07:44:57.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.815+0000 7fd8a8961640 1 --2- 172.21.15.47:0/3370361214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009e50 tx=0x7fd89c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9022139119df3a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 1 -- 172.21.15.47:0/3370361214 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c02ca70 con 0x7fd8a4144b80 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: dump: 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.824 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: _finish_auth 0 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 1 -- 172.21.15.47:0/3370361214 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c02cbd0 con 0x7fd8a4144b80 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8aa3c4640 5 monclient: authenticate success, global_id 17114 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8aa3c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8aa3c4640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:57.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8aa3c4640 4 set_mon_vals no callback set 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 1 -- 172.21.15.47:0/3370361214 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c0343f0 con 0x7fd8a4144b80 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.821+0000 7fd8a2ffd640 10 monclient: dump: 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.826 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 10 monclient: shutdown 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/3370361214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 msgr2=0x7fd8a4145010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/3370361214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7fd89c009e50 tx=0x7fd89c02bed0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/3370361214 shutdown_connections 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/3370361214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a4144b80 0x7fd8a4145010 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/3370361214 >> v2:172.21.15.47:3300/0 conn(0x7fd8a40b43c0 0x7fd8a40a8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/3370361214 >> 172.21.15.47:0/3370361214 conn(0x7fd8a4071930 msgr2=0x7fd8a4071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/3370361214 shutdown_connections 2024-03-30T07:44:57.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.822+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/3370361214 wait complete. 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.823+0000 7fd8aa3c4640 10 monclient: build_initial_monmap 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.823+0000 7fd8aa3c4640 10 monclient: monmap: 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:57.831 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.823+0000 7fd8aa3c4640 1 librados: starting msgr at 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.823+0000 7fd8aa3c4640 1 librados: starting objecter 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.823+0000 7fd8aa3c4640 1 Processor -- start 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 -- start start 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 librados: setting wanted keys 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 librados: calling monclient init 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient: init 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 0x7fd8a40b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd8a40b43c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:57.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a413e890 0x7fd8a413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd8a413e890 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a4144b80 0x7fd8a41d6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a4144b80 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41412e0 con 0x7fd8a40b43c0 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a41d6680 con 0x7fd8a4144b80 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4140090 con 0x7fd8a413e890 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.824+0000 7fd8aa3c4640 10 monclient(hunting): authenticate will time out at 3223.939927s 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 0x7fd8a40b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a37fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a413e890 0x7fd8a413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a8961640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a4144b80 0x7fd8a41d6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 0x7fd8a40b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54024/0 (socket says 172.21.15.47:54024) 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 1 -- 172.21.15.47:0/944117190 learned_addr learned my addr 172.21.15.47:0/944117190 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 10 monclient(hunting): get_auth_request con 0x7fd8a40b43c0 auth_method 0 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a37fe640 10 monclient(hunting): get_auth_request con 0x7fd8a413e890 auth_method 0 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a8961640 10 monclient(hunting): get_auth_request con 0x7fd8a4144b80 auth_method 0 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a8961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a8961640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.825+0000 7fd8a8961640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:57.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.826+0000 7fd8a8961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.828+0000 7fd8a8961640 10 monclient(hunting): handle_auth_done global_id 22396 payload 824 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 -- 172.21.15.47:0/944117190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a413e890 msgr2=0x7fd8a413fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8a413e890 0x7fd8a413fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 -- 172.21.15.47:0/944117190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 msgr2=0x7fd8a40b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 0x7fd8a40b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 10 monclient: _finish_hunting 0 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 monclient: found mon.noname-b 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a8961640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c009b30 con 0x7fd8a4144b80 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.829+0000 7fd8a3fff640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8a40b43c0 0x7fd8a40b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.831+0000 7fd8a8961640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a4144b80 0x7fd8a41d6ba0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4072590 tx=0x7fd89c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.831+0000 7fd8a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd8a4144b80 2024-03-30T07:44:57.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.831+0000 7fd8a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c040670 con 0x7fd8a4144b80 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: dump: 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.839 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: _finish_auth 0 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c0407d0 con 0x7fd8a4144b80 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c040bc0 con 0x7fd8a4144b80 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 5 monclient: authenticate success, global_id 22396 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 10 monclient: _renew_subs 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a40b6c10 con 0x7fd8a4144b80 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a27fc640 4 set_mon_vals no callback set 2024-03-30T07:44:57.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8a0ff9640 10 monclient: dump: 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 10 client.22396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 10 monclient: _renew_subs 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a40b5330 con 0x7fd8a4144b80 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.836+0000 7fd8aa3c4640 1 librados: init done 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 10 client.22396.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 15 client.22396.objecter _session_command_op_assign -1 1 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 20 client.22396.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 10 client.22396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:57.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.837+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8a4141450 con 0x7fd8a4144b80 2024-03-30T07:44:57.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.858+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd89c040d20 con 0x7fd8a4144b80 2024-03-30T07:44:57.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.858+0000 7fd8a0ff9640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd89006c5e0 0x7fd89006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.859+0000 7fd8a3fff640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd89006c5e0 0x7fd89006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.859+0000 7fd8a3fff640 10 monclient: get_auth_request con 0x7fd89006c5e0 auth_method 0 2024-03-30T07:44:57.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.859+0000 7fd8a3fff640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd89006c5e0 0x7fd89006eaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd8940112c0 tx=0x7fd894009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.859+0000 7fd8a0ff9640 10 client.22396.objecter ms_handle_connect 0x7fd89006c5e0 2024-03-30T07:44:57.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fd89c07b6d0 con 0x7fd8a4144b80 2024-03-30T07:44:57.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 10 client.22396.objecter ms_dispatch 0x7fd8a4000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 3 client.22396.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 3 client.22396.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 10 client.22396.objecter checking command 1 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7fd890073fb0 0x7fd890076450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter _get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 15 client.22396.objecter _session_command_op_remove -1 1 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter _get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 15 client.22396.objecter _session_command_op_assign 5 1 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 4 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 10 client.22396.objecter _send_command 1 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd890076b20 con 0x7fd890073fb0 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 20 client.22396.objecter dump_active .. 0 homeless 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd89c077a10 con 0x7fd8a4144b80 2024-03-30T07:44:57.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.862+0000 7fd8a0ff9640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:57.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.863+0000 7fd8a37fe640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7fd890073fb0 0x7fd890076450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:57.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.863+0000 7fd8a37fe640 10 monclient: get_auth_request con 0x7fd890073fb0 auth_method 0 2024-03-30T07:44:57.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.866+0000 7fd8a0ff9640 10 client.22396.objecter ms_handle_connect 0x7fd890073fb0 2024-03-30T07:44:57.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.866+0000 7fd8a37fe640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7fd890073fb0 0x7fd890076450 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.890+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== osd.5 v2:172.21.15.52:6826/35314 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd890076b20 con 0x7fd890073fb0 2024-03-30T07:44:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.890+0000 7fd8a0ff9640 10 client.22396.objecter ms_dispatch 0x7fd8a4000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.890+0000 7fd8a0ff9640 10 client.22396.objecter _finish_command 1 = system:0 2024-03-30T07:44:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.890+0000 7fd8a0ff9640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 4 2024-03-30T07:44:57.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.890+0000 7fd8a0ff9640 15 client.22396.objecter _session_command_op_remove 5 1 2024-03-30T07:44:57.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 10 client.22396.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:57.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 15 client.22396.objecter _session_command_op_assign -1 2 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 20 client.22396.objecter _get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 20 client.22396.objecter _get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 15 client.22396.objecter _session_command_op_remove -1 2 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 20 client.22396.objecter get_session s=0x7fd8900738e0 osd=5 3 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 15 client.22396.objecter _session_command_op_assign 5 2 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 4 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 10 client.22396.objecter _send_command 2 2024-03-30T07:44:57.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.902+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 --> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd8a414bf10 con 0x7fd890073fb0 2024-03-30T07:44:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.916+0000 7fd8a0ff9640 1 -- 172.21.15.47:0/944117190 <== osd.5 v2:172.21.15.52:6826/35314 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd898001440 con 0x7fd890073fb0 2024-03-30T07:44:57.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.916+0000 7fd8a0ff9640 10 client.22396.objecter ms_dispatch 0x7fd8a4000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.916+0000 7fd8a0ff9640 10 client.22396.objecter _finish_command 2 = system:0 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.916+0000 7fd8a0ff9640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 4 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.916+0000 7fd8a0ff9640 15 client.22396.objecter _session_command_op_remove 5 2 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:57.919 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:57.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 10 librados: watch_flush enter 2024-03-30T07:44:57.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 10 librados: watch_flush exit 2024-03-30T07:44:57.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 10 client.22396.objecter close_session for osd.5 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7fd890073fb0 msgr2=0x7fd890076450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6826/35314,v1:172.21.15.52:6827/35314] conn(0x7fd890073fb0 0x7fd890076450 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 20 client.22396.objecter put_session s=0x7fd8900738e0 osd=5 1 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 20 client.22396.objecter shutdown clearing up homeless session... 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 10 client.22396.objecter successfully canceled tick 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd89006c5e0 msgr2=0x7fd89006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fd89006c5e0 0x7fd89006eaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd8940112c0 tx=0x7fd894009190 comp rx=0 tx=0).stop 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 10 monclient: shutdown 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a4144b80 msgr2=0x7fd8a41d6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 --2- 172.21.15.47:0/944117190 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fd8a4144b80 0x7fd8a41d6ba0 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4072590 tx=0x7fd89c004a40 comp rx=0 tx=0).stop 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8a8961640 1 -- 172.21.15.47:0/944117190 reap_dead start 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 shutdown_connections 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 >> 172.21.15.47:0/944117190 conn(0x7fd8a4071930 msgr2=0x7fd8a413ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:57.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 shutdown_connections 2024-03-30T07:44:57.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 -- 172.21.15.47:0/944117190 wait complete. 2024-03-30T07:44:57.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:57.917+0000 7fd8aa3c4640 1 librados: shutdown 2024-03-30T07:44:57.931 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.034+0000 7fa434134640 10 monclient: get_monmap_and_config 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.034+0000 7fa434134640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.034+0000 7fa434134640 10 monclient: monmap: 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:58.035090+0000 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:58.035090+0000 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.036 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:58.037 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.034+0000 7fa434134640 1 Processor -- start 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 -- start start 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient: init 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa42c0b4280 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fa42c13e930 0x7fa42c13eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c13e930 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:58.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 0x7fa42c141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c141670 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c13f2e0 con 0x7fa42c0b4280 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c0b4660 con 0x7fa42c141670 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c0aa3c0 con 0x7fa42c13e930 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.035+0000 7fa434134640 10 monclient(hunting): authenticate will time out at 3224.150870s 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 0x7fa42c141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4316cf640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7fa42c13e930 0x7fa42c13eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 0x7fa42c141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54030/0 (socket says 172.21.15.47:54030) 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 1 -- 172.21.15.47:0/235058358 learned_addr learned my addr 172.21.15.47:0/235058358 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 10 monclient(hunting): get_auth_request con 0x7fa42c141670 auth_method 0 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4326d1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4316cf640 10 monclient(hunting): get_auth_request con 0x7fa42c13e930 auth_method 0 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4316cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4316cf640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.036+0000 7fa4316cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4316cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4316cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4316cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.037+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 10 monclient(hunting): handle_auth_done global_id 22399 payload 274 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 1 -- 172.21.15.47:0/235058358 >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 msgr2=0x7fa42c141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 1 --2- 172.21.15.47:0/235058358 >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 0x7fa42c141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 1 -- 172.21.15.47:0/235058358 >> v1:172.21.15.47:6789/0 conn(0x7fa42c0b4280 legacy=0x7fa42c0a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 1 monclient: found mon.noname-c 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.038+0000 7fa4316cf640 1 -- 172.21.15.47:0/235058358 --> v2:172.21.15.52:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0aa630 con 0x7fa42c13e930 2024-03-30T07:44:58.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.040+0000 7fa4316cf640 1 --2- 172.21.15.47:0/235058358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c13e930 0x7fa42c13eda0 secure :-1 s=READY pgs=5527 cs=0 l=1 rev1=1 crypto rx=0x7fa42c071320 tx=0x7fa41c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c2239342532b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 1 -- 172.21.15.47:0/235058358 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c02e070 con 0x7fa42c13e930 2024-03-30T07:44:58.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:44:58.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: dump: 2024-03-30T07:44:58.046 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 1 -- 172.21.15.47:0/235058358 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c0315a0 con 0x7fa42c13e930 2024-03-30T07:44:58.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa434134640 5 monclient: authenticate success, global_id 22399 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa434134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa434134640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa434134640 4 set_mon_vals no callback set 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 1 -- 172.21.15.47:0/235058358 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c031930 con 0x7fa42c13e930 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.044+0000 7fa430ece640 10 monclient: dump: 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.048 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.049 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.049 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.049 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 10 monclient: shutdown 2024-03-30T07:44:58.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 -- 172.21.15.47:0/235058358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c13e930 msgr2=0x7fa42c13eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 --2- 172.21.15.47:0/235058358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c13e930 0x7fa42c13eda0 secure :-1 s=READY pgs=5527 cs=0 l=1 rev1=1 crypto rx=0x7fa42c071320 tx=0x7fa41c02b840 comp rx=0 tx=0).stop 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 -- 172.21.15.47:0/235058358 shutdown_connections 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 --2- 172.21.15.47:0/235058358 >> v2:172.21.15.47:3300/0 conn(0x7fa42c141670 0x7fa42c141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 --2- 172.21.15.47:0/235058358 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c13e930 0x7fa42c13eda0 unknown :-1 s=CLOSED pgs=5527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 -- 172.21.15.47:0/235058358 >> 172.21.15.47:0/235058358 conn(0x7fa42c065e00 msgr2=0x7fa42c066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 -- 172.21.15.47:0/235058358 shutdown_connections 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.045+0000 7fa434134640 1 -- 172.21.15.47:0/235058358 wait complete. 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 10 monclient: monmap: 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:58.052 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 librados: starting msgr at 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 librados: starting objecter 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 Processor -- start 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 -- start start 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 librados: setting wanted keys 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 1 librados: calling monclient init 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.046+0000 7fa434134640 10 monclient: init 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c0b4280 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa42c13e930 0x7fa42c1d64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c13e930 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c141670 0x7fa42c12c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): picked mon.noname-b con 0x7fa42c141670 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c134cc0 con 0x7fa42c0b4280 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c1d7330 con 0x7fa42c141670 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c1d6e10 con 0x7fa42c13e930 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa434134640 10 monclient(hunting): authenticate will time out at 3224.162472s 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.047+0000 7fa431ed0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54038/0 (socket says 172.21.15.47:54038) 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 1 -- 172.21.15.47:0/1664669725 learned_addr learned my addr 172.21.15.47:0/1664669725 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): get_auth_request con 0x7fa42c0b4280 auth_method 0 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.048+0000 7fa431ed0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.049+0000 7fa4326d1640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c141670 0x7fa42c12c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): get_auth_request con 0x7fa42c141670 auth_method 0 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.050+0000 7fa4326d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 10 monclient(hunting): handle_auth_done global_id 15903 payload 824 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 -- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa42c13e930 msgr2=0x7fa42c1d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa42c13e930 0x7fa42c1d64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 -- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c141670 msgr2=0x7fa42c12c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c141670 0x7fa42c12c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 monclient: found mon.noname-a 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:58.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.051+0000 7fa431ed0640 1 -- 172.21.15.47:0/1664669725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c009400 con 0x7fa42c0b4280 2024-03-30T07:44:58.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.056+0000 7fa431ed0640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c4d0 tx=0x7fa42800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.056+0000 7fa4227fc640 10 client.?.objecter ms_handle_connect 0x7fa42c0b4280 2024-03-30T07:44:58.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.056+0000 7fa4227fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280052d0 con 0x7fa42c0b4280 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: dump: 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.080 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa428005430 con 0x7fa42c0b4280 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 5 monclient: authenticate success, global_id 15903 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 10 monclient: _renew_subs 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c132a90 con 0x7fa42c0b4280 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428012850 con 0x7fa42c0b4280 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa423fff640 4 set_mon_vals no callback set 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa4227fc640 10 monclient: dump: 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.082 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 10 client.15903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 10 monclient: _renew_subs 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c1d6bd0 con 0x7fa42c0b4280 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.077+0000 7fa434134640 1 librados: init done 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.078+0000 7fa434134640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.078+0000 7fa434134640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.078+0000 7fa434134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:58.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.078+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c04e030 con 0x7fa42c0b4280 2024-03-30T07:44:58.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.345+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa4280129b0 con 0x7fa42c0b4280 2024-03-30T07:44:58.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.346+0000 7fa4227fc640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 0x7fa41006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.346+0000 7fa4316cf640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 0x7fa41006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.346+0000 7fa4316cf640 10 monclient: get_auth_request con 0x7fa41006c5e0 auth_method 0 2024-03-30T07:44:58.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.347+0000 7fa4227fc640 10 client.15903.objecter ms_handle_connect 0x7fa41006c5e0 2024-03-30T07:44:58.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.347+0000 7fa4316cf640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 0x7fa41006eaa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa41c036000 tx=0x7fa41c030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7fa428058c40 con 0x7fa42c0b4280 2024-03-30T07:44:58.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 10 client.15903.objecter ms_dispatch 0x7fa42c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 3 client.15903.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 3 client.15903.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 20 client.15903.objecter dump_active .. 0 homeless 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7fa428054880 con 0x7fa42c0b4280 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.360+0000 7fa4227fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:58.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.568+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fa42c1d6bd0 con 0x7fa41006c5e0 2024-03-30T07:44:58.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.569+0000 7fa4227fc640 1 -- 172.21.15.47:0/1664669725 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa42c1d6bd0 con 0x7fa41006c5e0 2024-03-30T07:44:58.571 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 5 to deep-scrub 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 10 librados: watch_flush enter 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 10 librados: watch_flush exit 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 20 client.15903.objecter shutdown clearing up homeless session... 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 10 client.15903.objecter successfully canceled tick 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 msgr2=0x7fa41006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 0x7fa41006eaa0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa41c036000 tx=0x7fa41c030000 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 10 monclient: shutdown 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 msgr2=0x7fa42c140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c4d0 tx=0x7fa42800c990 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 shutdown_connections 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7fa41006c5e0 0x7fa41006eaa0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7fa42c141670 0x7fa42c12c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa42c13e930 0x7fa42c1d64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 --2- 172.21.15.47:0/1664669725 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa42c0b4280 0x7fa42c140f00 unknown :-1 s=CLOSED pgs=3149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 >> 172.21.15.47:0/1664669725 conn(0x7fa42c065e00 msgr2=0x7fa42c0b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 shutdown_connections 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 -- 172.21.15.47:0/1664669725 wait complete. 2024-03-30T07:44:58.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.571+0000 7fa434134640 1 librados: shutdown 2024-03-30T07:44:58.583 INFO:tasks.ceph:Scrubbing osd.6 2024-03-30T07:44:58.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:58.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.686+0000 7f8c54b37640 10 monclient: get_monmap_and_config 2024-03-30T07:44:58.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.686+0000 7f8c54b37640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.686+0000 7f8c54b37640 10 monclient: monmap: 2024-03-30T07:44:58.688 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:58.687228+0000 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:58.687228+0000 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:58.689 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 Processor -- start 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 -- start start 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient: init 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 0x7f8c50144a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-b con 0x7f8c500b4320 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f8c50144fa0 0x7f8c5013e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-c con 0x7f8c50144fa0 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c500a8fc0 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c500a94e0 con 0x7f8c500a8fc0 2024-03-30T07:44:58.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c500b4700 con 0x7f8c50144fa0 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c500aa350 con 0x7f8c500b4320 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.687+0000 7f8c54b37640 10 monclient(hunting): authenticate will time out at 3224.803013s 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4e59c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 0x7f8c50144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c47fff640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f8c50144fa0 0x7f8c5013e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4ed9d640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8c500a8fc0 0x7f8c500b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33238/0 (socket says 172.21.15.47:33238) 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/1951239049 learned_addr learned my addr 172.21.15.47:0/1951239049 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4e59c640 10 monclient(hunting): get_auth_request con 0x7f8c500b4320 auth_method 0 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4e59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c47fff640 10 monclient(hunting): get_auth_request con 0x7f8c50144fa0 auth_method 0 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c47fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.688+0000 7f8c47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.689+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_done global_id 17120 payload 274 2024-03-30T07:44:58.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 1 -- 172.21.15.47:0/1951239049 >> v2:172.21.15.52:3300/0 conn(0x7f8c50144fa0 msgr2=0x7f8c5013e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 1 --2- 172.21.15.47:0/1951239049 >> v2:172.21.15.52:3300/0 conn(0x7f8c50144fa0 0x7f8c5013e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 1 -- 172.21.15.47:0/1951239049 >> v1:172.21.15.47:6789/0 conn(0x7f8c500a8fc0 legacy=0x7f8c500b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 1 monclient: found mon.noname-b 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:58.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.691+0000 7f8c4e59c640 1 -- 172.21.15.47:0/1951239049 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500a8aa0 con 0x7f8c500b4320 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.693+0000 7f8c4e59c640 1 --2- 172.21.15.47:0/1951239049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 0x7f8c50144a60 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f8c400097a0 tx=0x7f8c4002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8fbb4ca06490f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 1 -- 172.21.15.47:0/1951239049 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4002e070 con 0x7f8c500b4320 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 10 monclient: dump: 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.697 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c54b37640 5 monclient: authenticate success, global_id 17120 2024-03-30T07:44:58.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.695+0000 7f8c54b37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.696+0000 7f8c4dd9b640 1 -- 172.21.15.47:0/1951239049 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c400315a0 con 0x7f8c500b4320 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.696+0000 7f8c4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.696+0000 7f8c54b37640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.696+0000 7f8c54b37640 4 set_mon_vals no callback set 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c4dd9b640 1 -- 172.21.15.47:0/1951239049 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c40031930 con 0x7f8c500b4320 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c4dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c4dd9b640 10 monclient: dump: 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 10 monclient: shutdown 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 -- 172.21.15.47:0/1951239049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 msgr2=0x7f8c50144a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 --2- 172.21.15.47:0/1951239049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 0x7f8c50144a60 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f8c400097a0 tx=0x7f8c4002b840 comp rx=0 tx=0).stop 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 -- 172.21.15.47:0/1951239049 shutdown_connections 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 --2- 172.21.15.47:0/1951239049 >> v2:172.21.15.52:3300/0 conn(0x7f8c50144fa0 0x7f8c5013e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 --2- 172.21.15.47:0/1951239049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500b4320 0x7f8c50144a60 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 -- 172.21.15.47:0/1951239049 >> 172.21.15.47:0/1951239049 conn(0x7f8c50065e00 msgr2=0x7f8c50066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 -- 172.21.15.47:0/1951239049 shutdown_connections 2024-03-30T07:44:58.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.697+0000 7f8c54b37640 1 -- 172.21.15.47:0/1951239049 wait complete. 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.698+0000 7f8c54b37640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.698+0000 7f8c54b37640 10 monclient: monmap: 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:58.703 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 librados: starting msgr at 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 librados: starting objecter 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 Processor -- start 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 -- start start 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 librados: setting wanted keys 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 librados: calling monclient init 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient: init 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500a8fc0 0x7f8c500abb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-c con 0x7f8c500a8fc0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:58.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c500b4320 0x7f8c501d6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-a con 0x7f8c500b4320 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 0x7f8c501d67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): picked mon.noname-b con 0x7f8c50144fa0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c500ac3b0 con 0x7f8c500b4320 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c500ac040 con 0x7f8c50144fa0 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c50141d50 con 0x7f8c500a8fc0 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.699+0000 7f8c54b37640 10 monclient(hunting): authenticate will time out at 3224.814946s 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 0x7f8c501d67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c47fff640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c500b4320 0x7f8c501d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4e59c640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500a8fc0 0x7f8c500abb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 0x7f8c501d67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:52186/0 (socket says 172.21.15.47:52186) 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/69490991 learned_addr learned my addr 172.21.15.47:0/69490991 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 10 monclient(hunting): get_auth_request con 0x7f8c50144fa0 auth_method 0 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4e59c640 10 monclient(hunting): get_auth_request con 0x7f8c500a8fc0 auth_method 0 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4e59c640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.700+0000 7f8c4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): get_auth_request con 0x7f8c500b4320 auth_method 0 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.701+0000 7f8c47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 10 monclient(hunting): handle_auth_done global_id 22408 payload 824 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/69490991 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500a8fc0 msgr2=0x7f8c500abb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8c500a8fc0 0x7f8c500abb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/69490991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c500b4320 msgr2=0x7f8c501d6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8c500b4320 0x7f8c501d6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 monclient: found mon.noname-b 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.703+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c40009400 con 0x7f8c50144fa0 2024-03-30T07:44:58.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.708+0000 7f8c4ed9d640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 0x7f8c501d67b0 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800cbc0 tx=0x7f8c4800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.708+0000 7f8c46ffd640 10 client.?.objecter ms_handle_connect 0x7f8c50144fa0 2024-03-30T07:44:58.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.708+0000 7f8c46ffd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800ad40 con 0x7f8c50144fa0 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: dump: 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.715 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c48012400 con 0x7f8c50144fa0 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4800aea0 con 0x7f8c50144fa0 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c46ffd640 10 monclient: dump: 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 5 monclient: authenticate success, global_id 22408 2024-03-30T07:44:58.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 10 monclient: _renew_subs 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c4d59a640 4 set_mon_vals no callback set 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c500b1890 con 0x7f8c50144fa0 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 10 client.22408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 10 monclient: _renew_subs 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c500aaf10 con 0x7f8c50144fa0 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.712+0000 7f8c54b37640 1 librados: init done 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 10 client.22408.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 15 client.22408.objecter _session_command_op_assign -1 1 2024-03-30T07:44:58.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 20 client.22408.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:58.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 10 client.22408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:58.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.713+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c500b1710 con 0x7f8c50144fa0 2024-03-30T07:44:58.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.733+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c48007070 con 0x7f8c50144fa0 2024-03-30T07:44:58.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.734+0000 7f8c46ffd640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f8c3006c590 0x7f8c3006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.734+0000 7f8c4e59c640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f8c3006c590 0x7f8c3006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.734+0000 7f8c4e59c640 10 monclient: get_auth_request con 0x7f8c3006c590 auth_method 0 2024-03-30T07:44:58.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.734+0000 7f8c46ffd640 10 client.22408.objecter ms_handle_connect 0x7f8c3006c590 2024-03-30T07:44:58.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.734+0000 7f8c4e59c640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f8c3006c590 0x7f8c3006ea50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8c40036000 tx=0x7f8c4003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f8c48058980 con 0x7f8c50144fa0 2024-03-30T07:44:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 10 client.22408.objecter ms_dispatch 0x7f8c50000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 3 client.22408.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 3 client.22408.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:58.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 10 client.22408.objecter checking command 1 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f8c30073f60 0x7f8c30076400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter _get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 15 client.22408.objecter _session_command_op_remove -1 1 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter _get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 15 client.22408.objecter _session_command_op_assign 6 1 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 4 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 10 client.22408.objecter _send_command 1 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c30076ad0 con 0x7f8c30073f60 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 20 client.22408.objecter dump_active .. 0 homeless 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c48001c00 con 0x7f8c50144fa0 2024-03-30T07:44:58.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c46ffd640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:58.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.737+0000 7f8c47fff640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f8c30073f60 0x7f8c30076400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.738+0000 7f8c47fff640 10 monclient: get_auth_request con 0x7f8c30073f60 auth_method 0 2024-03-30T07:44:58.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.742+0000 7f8c47fff640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f8c30073f60 0x7f8c30076400 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.742+0000 7f8c46ffd640 10 client.22408.objecter ms_handle_connect 0x7f8c30073f60 2024-03-30T07:44:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.766+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== osd.6 v2:172.21.15.52:6802/35315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f8c30076ad0 con 0x7f8c30073f60 2024-03-30T07:44:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.766+0000 7f8c46ffd640 10 client.22408.objecter ms_dispatch 0x7f8c50000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.766+0000 7f8c46ffd640 10 client.22408.objecter _finish_command 1 = system:0 2024-03-30T07:44:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.766+0000 7f8c46ffd640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 4 2024-03-30T07:44:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.766+0000 7f8c46ffd640 15 client.22408.objecter _session_command_op_remove 6 1 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 10 client.22408.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 15 client.22408.objecter _session_command_op_assign -1 2 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 20 client.22408.objecter _get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 20 client.22408.objecter _get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 15 client.22408.objecter _session_command_op_remove -1 2 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 20 client.22408.objecter get_session s=0x7f8c30073890 osd=6 3 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 15 client.22408.objecter _session_command_op_assign 6 2 2024-03-30T07:44:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 4 2024-03-30T07:44:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 10 client.22408.objecter _send_command 2 2024-03-30T07:44:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.779+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 --> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8c5013ecd0 con 0x7f8c30073f60 2024-03-30T07:44:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c46ffd640 1 -- 172.21.15.47:0/69490991 <== osd.6 v2:172.21.15.52:6802/35315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8c38009bd0 con 0x7f8c30073f60 2024-03-30T07:44:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c46ffd640 10 client.22408.objecter ms_dispatch 0x7f8c50000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c46ffd640 10 client.22408.objecter _finish_command 2 = system:0 2024-03-30T07:44:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c46ffd640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 4 2024-03-30T07:44:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c46ffd640 15 client.22408.objecter _session_command_op_remove 6 2 2024-03-30T07:44:58.796 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:58.796 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:58.796 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:58.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 10 librados: watch_flush enter 2024-03-30T07:44:58.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 10 librados: watch_flush exit 2024-03-30T07:44:58.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 10 client.22408.objecter close_session for osd.6 2024-03-30T07:44:58.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f8c30073f60 msgr2=0x7f8c30076400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6802/35315,v1:172.21.15.52:6803/35315] conn(0x7f8c30073f60 0x7f8c30076400 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 20 client.22408.objecter put_session s=0x7f8c30073890 osd=6 1 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 20 client.22408.objecter shutdown clearing up homeless session... 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 10 client.22408.objecter successfully canceled tick 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f8c3006c590 msgr2=0x7f8c3006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f8c3006c590 0x7f8c3006ea50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8c40036000 tx=0x7f8c4003d040 comp rx=0 tx=0).stop 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 10 monclient: shutdown 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 msgr2=0x7f8c501d67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 --2- 172.21.15.47:0/69490991 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f8c50144fa0 0x7f8c501d67b0 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800cbc0 tx=0x7f8c4800a430 comp rx=0 tx=0).stop 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c4ed9d640 1 -- 172.21.15.47:0/69490991 reap_dead start 2024-03-30T07:44:58.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 shutdown_connections 2024-03-30T07:44:58.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 >> 172.21.15.47:0/69490991 conn(0x7f8c50065e00 msgr2=0x7f8c50068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:58.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 shutdown_connections 2024-03-30T07:44:58.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 -- 172.21.15.47:0/69490991 wait complete. 2024-03-30T07:44:58.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.793+0000 7f8c54b37640 1 librados: shutdown 2024-03-30T07:44:58.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-30T07:44:58.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.906+0000 7f085aa51640 10 monclient: get_monmap_and_config 2024-03-30T07:44:58.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.906+0000 7f085aa51640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.906+0000 7f085aa51640 10 monclient: monmap: 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:58.907273+0000 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:58.907273+0000 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:58.909 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:58.910 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.907+0000 7f085aa51640 1 Processor -- start 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.907+0000 7f085aa51640 1 -- start start 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.907+0000 7f085aa51640 10 monclient: init 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f085406fd20 0x7f08540afb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-c con 0x7f085406fd20 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0854070100 addr v1:172.21.15.47:6789/0 2024-03-30T07:44:58.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 0x7f085413ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-b con 0x7f085413c5b0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854072070 con 0x7f0854070100 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085413cf40 con 0x7f085406fd20 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08541401f0 con 0x7f085413c5b0 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f085aa51640 10 monclient(hunting): authenticate will time out at 3225.023272s 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f08537fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0854070100 0x7f085413fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33242/0 (socket says 172.21.15.47:33242) 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f0858fee640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 0x7f085413ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f08537fe640 1 -- 172.21.15.47:0/791182214 learned_addr learned my addr 172.21.15.47:0/791182214 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.908+0000 7f0853fff640 1 --2- 172.21.15.47:0/791182214 >> v2:172.21.15.52:3300/0 conn(0x7f085406fd20 0x7f08540afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): get_auth_request con 0x7f085413c5b0 auth_method 0 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): get_auth_request con 0x7f085406fd20 auth_method 0 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.909+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 10 monclient(hunting): handle_auth_done global_id 17126 payload 274 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 1 -- 172.21.15.47:0/791182214 >> v2:172.21.15.52:3300/0 conn(0x7f085406fd20 msgr2=0x7f08540afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 1 --2- 172.21.15.47:0/791182214 >> v2:172.21.15.52:3300/0 conn(0x7f085406fd20 0x7f08540afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 1 -- 172.21.15.47:0/791182214 >> v1:172.21.15.47:6789/0 conn(0x7f0854070100 legacy=0x7f085413fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 1 monclient: found mon.noname-b 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:58.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.911+0000 7f0858fee640 1 -- 172.21.15.47:0/791182214 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0854070f80 con 0x7f085413c5b0 2024-03-30T07:44:58.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.913+0000 7f0858fee640 1 --2- 172.21.15.47:0/791182214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 0x7f085413ca00 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f084c009d90 tx=0x7f084c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=121ac86c704dc916 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 1 -- 172.21.15.47:0/791182214 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c02cbc0 con 0x7f085413c5b0 2024-03-30T07:44:58.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: dump: 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 1 -- 172.21.15.47:0/791182214 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084c02cd20 con 0x7f085413c5b0 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f085aa51640 5 monclient: authenticate success, global_id 17126 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f085aa51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f085aa51640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 1 -- 172.21.15.47:0/791182214 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084c034660 con 0x7f085413c5b0 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f085aa51640 4 set_mon_vals no callback set 2024-03-30T07:44:58.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.916+0000 7f0852ffd640 10 monclient: dump: 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.921 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.922 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 10 monclient: shutdown 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 -- 172.21.15.47:0/791182214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 msgr2=0x7f085413ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 --2- 172.21.15.47:0/791182214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 0x7f085413ca00 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f084c009d90 tx=0x7f084c02be10 comp rx=0 tx=0).stop 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 -- 172.21.15.47:0/791182214 shutdown_connections 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 --2- 172.21.15.47:0/791182214 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f085413c5b0 0x7f085413ca00 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 --2- 172.21.15.47:0/791182214 >> v2:172.21.15.52:3300/0 conn(0x7f085406fd20 0x7f08540afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 -- 172.21.15.47:0/791182214 >> 172.21.15.47:0/791182214 conn(0x7f08540b5230 msgr2=0x7f08540b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 -- 172.21.15.47:0/791182214 shutdown_connections 2024-03-30T07:44:58.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.917+0000 7f085aa51640 1 -- 172.21.15.47:0/791182214 wait complete. 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 10 monclient: build_initial_monmap 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 10 monclient: monmap: 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 librados: starting msgr at 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 librados: starting objecter 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 Processor -- start 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 -- start start 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 librados: setting wanted keys 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 1 librados: calling monclient init 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.918+0000 7f085aa51640 10 monclient: init 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-b con 0x7f085406fd20 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0854070100 0x7f08541d7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-c con 0x7f0854070100 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f085413c5b0 0x7f08541d7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): picked mon.noname-a con 0x7f085413c5b0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0854133e30 con 0x7f085413c5b0 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0854137530 con 0x7f085406fd20 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08541d82f0 con 0x7f0854070100 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f085aa51640 10 monclient(hunting): authenticate will time out at 3225.034614s 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f08537fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0854070100 0x7f08541d7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.919+0000 7f0853fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0858fee640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f085413c5b0 0x7f08541d7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:52212/0 (socket says 172.21.15.47:52212) 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 1 -- 172.21.15.47:0/2460963055 learned_addr learned my addr 172.21.15.47:0/2460963055 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): get_auth_request con 0x7f085406fd20 auth_method 0 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): get_auth_request con 0x7f0854070100 auth_method 0 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0858fee640 10 monclient(hunting): get_auth_request con 0x7f085413c5b0 auth_method 0 2024-03-30T07:44:58.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0858fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0858fee640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0858fee640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f0853fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.920+0000 7f08537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.921+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.921+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.921+0000 7f0858fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:58.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 10 monclient(hunting): handle_auth_done global_id 22414 payload 824 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 -- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0854070100 msgr2=0x7f08541d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0854070100 0x7f08541d7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 -- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f085413c5b0 msgr2=0x7f08541d7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f085413c5b0 0x7f08541d7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 10 monclient: _finish_hunting 0 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 monclient: found mon.noname-b 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.923+0000 7f0853fff640 1 -- 172.21.15.47:0/2460963055 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c009a70 con 0x7f085406fd20 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.925+0000 7f0853fff640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f084400e9b0 tx=0x7f084400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.925+0000 7f0850ff9640 10 client.?.objecter ms_handle_connect 0x7f085406fd20 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.925+0000 7f0850ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844009670 con 0x7f085406fd20 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: dump: 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.934 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: _finish_auth 0 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400c9a0 con 0x7f085406fd20 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400cd50 con 0x7f085406fd20 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f0850ff9640 10 monclient: dump: 2024-03-30T07:44:58.935 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 5 monclient: authenticate success, global_id 22414 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 10 monclient: _renew_subs 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0854136f80 con 0x7f085406fd20 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f08527fc640 4 set_mon_vals no callback set 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 10 client.22414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 10 monclient: _renew_subs 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0854136a30 con 0x7f085406fd20 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.929+0000 7f085aa51640 1 librados: init done 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.930+0000 7f085aa51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.930+0000 7f085aa51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.930+0000 7f085aa51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:58.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.930+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08541376f0 con 0x7f085406fd20 2024-03-30T07:44:58.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.950+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0844013470 con 0x7f085406fd20 2024-03-30T07:44:58.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.950+0000 7f0850ff9640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 0x7f083406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:58.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.950+0000 7f08537fe640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 0x7f083406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:58.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.950+0000 7f08537fe640 10 monclient: get_auth_request con 0x7f083406c5e0 auth_method 0 2024-03-30T07:44:58.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.951+0000 7f08537fe640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 0x7f083406eaa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f085413cb20 tx=0x7f083c007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:58.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:58.951+0000 7f0850ff9640 10 client.22414.objecter ms_handle_connect 0x7f083406c5e0 2024-03-30T07:44:59.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.208+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f0844058c00 con 0x7f085406fd20 2024-03-30T07:44:59.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.208+0000 7f0850ff9640 10 client.22414.objecter ms_dispatch 0x7f0854000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:59.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.208+0000 7f0850ff9640 3 client.22414.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:59.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.208+0000 7f0850ff9640 3 client.22414.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:59.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.208+0000 7f0850ff9640 20 client.22414.objecter dump_active .. 0 homeless 2024-03-30T07:44:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.218+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f0844054f40 con 0x7f085406fd20 2024-03-30T07:44:59.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.218+0000 7f0850ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:59.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.218+0000 7f0850ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:44:59.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.401+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f0854070b30 con 0x7f083406c5e0 2024-03-30T07:44:59.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.401+0000 7f0850ff9640 1 -- 172.21.15.47:0/2460963055 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f083c001460 con 0x7f083406c5e0 2024-03-30T07:44:59.403 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 6 to deep-scrub 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 10 librados: watch_flush enter 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 10 librados: watch_flush exit 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 20 client.22414.objecter shutdown clearing up homeless session... 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 10 client.22414.objecter successfully canceled tick 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 msgr2=0x7f083406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 0x7f083406eaa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f085413cb20 tx=0x7f083c007e80 comp rx=0 tx=0).stop 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 10 monclient: shutdown 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 msgr2=0x7f0854133270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f084400e9b0 tx=0x7f084400ee70 comp rx=0 tx=0).stop 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 shutdown_connections 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f083406c5e0 0x7f083406eaa0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f085413c5b0 0x7f08541d7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0854070100 0x7f08541d7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 --2- 172.21.15.47:0/2460963055 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f085406fd20 0x7f0854133270 unknown :-1 s=CLOSED pgs=5529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 >> 172.21.15.47:0/2460963055 conn(0x7f08540b5230 msgr2=0x7f08541346f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 shutdown_connections 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 -- 172.21.15.47:0/2460963055 wait complete. 2024-03-30T07:44:59.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.404+0000 7f085aa51640 1 librados: shutdown 2024-03-30T07:44:59.421 INFO:tasks.ceph:Scrubbing osd.7 2024-03-30T07:44:59.421 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.529+0000 7f90668b6640 10 monclient: get_monmap_and_config 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.529+0000 7f90668b6640 10 monclient: build_initial_monmap 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.529+0000 7f90668b6640 10 monclient: monmap: 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:59.530046+0000 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:59.530046+0000 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:59.532 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 Processor -- start 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 -- start start 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient: init 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f90600b4220 0x7f90600a8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-a con 0x7f90600b4220 addr v2:172.21.15.47:3300/0 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f906013eb50 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013efe0 0x7f9060141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-b con 0x7f906013efe0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90600a90d0 con 0x7f906013eb50 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600b4600 con 0x7f90600b4220 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600aa660 con 0x7f906013efe0 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.530+0000 7f90668b6640 10 monclient(hunting): authenticate will time out at 3225.645887s 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f90600b4220 0x7f90600a8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f90600b4220 0x7f90600a8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54068/0 (socket says 172.21.15.47:54068) 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 1 -- 172.21.15.47:0/174359867 learned_addr learned my addr 172.21.15.47:0/174359867 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 10 monclient(hunting): get_auth_request con 0x7f90600b4220 auth_method 0 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.531+0000 7f905ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.532+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.532+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.532+0000 7f905ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 10 monclient(hunting): handle_auth_done global_id 15918 payload 274 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 1 -- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013efe0 msgr2=0x7f9060141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 1 --2- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013efe0 0x7f9060141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 1 -- 172.21.15.47:0/174359867 >> v1:172.21.15.52:6789/0 conn(0x7f906013eb50 legacy=0x7f90601415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:59.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 10 monclient: _finish_hunting 0 2024-03-30T07:44:59.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 1 monclient: found mon.noname-a 2024-03-30T07:44:59.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:59.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:59.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.534+0000 7f905ffff640 1 -- 172.21.15.47:0/174359867 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600a9560 con 0x7f90600b4220 2024-03-30T07:44:59.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.537+0000 7f905ffff640 1 --2- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90600b4220 0x7f90600a8970 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f9050009870 tx=0x7f905002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc237100c5159f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 1 -- 172.21.15.47:0/174359867 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050031740 con 0x7f90600b4220 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: dump: 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.543 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: _finish_auth 0 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: _check_auth_tickets 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 1 -- 172.21.15.47:0/174359867 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90500318a0 con 0x7f90600b4220 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 1 -- 172.21.15.47:0/174359867 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050031c30 con 0x7f90600b4220 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f90668b6640 5 monclient: authenticate success, global_id 15918 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f90668b6640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f90668b6640 4 set_mon_vals no callback set 2024-03-30T07:44:59.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.541+0000 7f905effd640 10 monclient: dump: 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.545 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 10 monclient: shutdown 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 -- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90600b4220 msgr2=0x7f90600a8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 --2- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90600b4220 0x7f90600a8970 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f9050009870 tx=0x7f905002b7e0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 -- 172.21.15.47:0/174359867 shutdown_connections 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 --2- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013efe0 0x7f9060141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 --2- 172.21.15.47:0/174359867 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f90600b4220 0x7f90600a8970 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 -- 172.21.15.47:0/174359867 >> 172.21.15.47:0/174359867 conn(0x7f9060065bf0 msgr2=0x7f9060066000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 -- 172.21.15.47:0/174359867 shutdown_connections 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.542+0000 7f90668b6640 1 -- 172.21.15.47:0/174359867 wait complete. 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 10 monclient: build_initial_monmap 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 10 monclient: monmap: 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:59.549 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 librados: starting msgr at 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 librados: starting objecter 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 Processor -- start 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 -- start start 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 librados: setting wanted keys 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 1 librados: calling monclient init 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.543+0000 7f90668b6640 10 monclient: init 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:59.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90600b4220 0x7f9060138b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-b con 0x7f90600b4220 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013eb50 0x7f9060141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-c con 0x7f906013eb50 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 0x7f9060131410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): picked mon.noname-a con 0x7f9060141d60 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90601422c0 con 0x7f9060141d60 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906013e280 con 0x7f90600b4220 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906013e3f0 con 0x7f906013eb50 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:59.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f90668b6640 10 monclient(hunting): authenticate will time out at 3225.659397s 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.544+0000 7f9064e53640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 0x7f9060131410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 0x7f9060131410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54080/0 (socket says 172.21.15.47:54080) 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 1 -- 172.21.15.47:0/1464519951 learned_addr learned my addr 172.21.15.47:0/1464519951 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): get_auth_request con 0x7f9060141d60 auth_method 0 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.545+0000 7f9064e53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 10 monclient(hunting): handle_auth_done global_id 15921 payload 824 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 -- 172.21.15.47:0/1464519951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013eb50 msgr2=0x7f9060141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f906013eb50 0x7f9060141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 -- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90600b4220 msgr2=0x7f9060138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f90600b4220 0x7f9060138b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 10 monclient: _finish_hunting 0 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 monclient: found mon.noname-a 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:44:59.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.550+0000 7f9064e53640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90500094d0 con 0x7f9060141d60 2024-03-30T07:44:59.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.553+0000 7f9064e53640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 0x7f9060131410 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f905800b970 tx=0x7f905800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.553+0000 7f905cff9640 10 client.?.objecter ms_handle_connect 0x7f9060141d60 2024-03-30T07:44:59.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.553+0000 7f905cff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:59.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800cb40 con 0x7f9060141d60 2024-03-30T07:44:59.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:44:59.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: dump: 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: _finish_auth 0 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: _check_auth_tickets 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905800ccc0 con 0x7f9060141d60 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058014680 con 0x7f9060141d60 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905cff9640 10 monclient: dump: 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.562 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 5 monclient: authenticate success, global_id 15921 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 10 monclient: _renew_subs 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9060131950 con 0x7f9060141d60 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f905e7fc640 4 set_mon_vals no callback set 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 10 client.15921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 10 monclient: _renew_subs 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9060131de0 con 0x7f9060141d60 2024-03-30T07:44:59.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.558+0000 7f90668b6640 1 librados: init done 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 10 client.15921.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 15 client.15921.objecter _session_command_op_assign -1 1 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 20 client.15921.objecter _get_session osd=-1 returning homeless 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 10 client.15921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:44:59.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.559+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9060067480 con 0x7f9060141d60 2024-03-30T07:44:59.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.581+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f905801d3b0 con 0x7f9060141d60 2024-03-30T07:44:59.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.581+0000 7f905cff9640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f904c06c590 0x7f904c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.582+0000 7f905ffff640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f904c06c590 0x7f904c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.582+0000 7f905ffff640 10 monclient: get_auth_request con 0x7f904c06c590 auth_method 0 2024-03-30T07:44:59.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.582+0000 7f905cff9640 10 client.15921.objecter ms_handle_connect 0x7f904c06c590 2024-03-30T07:44:59.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.582+0000 7f905ffff640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f904c06c590 0x7f904c06ea50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9050035000 tx=0x7f9050030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f905800ce30 con 0x7f9060141d60 2024-03-30T07:44:59.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 10 client.15921.objecter ms_dispatch 0x7f9060000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:44:59.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 3 client.15921.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:44:59.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 3 client.15921.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:44:59.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 10 client.15921.objecter checking command 1 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f904c073f60 0x7f904c076400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter _get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 15 client.15921.objecter _session_command_op_remove -1 1 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter _get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 15 client.15921.objecter _session_command_op_assign 7 1 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 4 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 10 client.15921.objecter _send_command 1 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f904c076ad0 con 0x7f904c073f60 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 20 client.15921.objecter dump_active .. 0 homeless 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_get_version_reply(handle=1 version=994) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9058054be0 con 0x7f9060141d60 2024-03-30T07:44:59.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.586+0000 7f905cff9640 10 monclient: handle_get_version_reply finishing 1 version 994 2024-03-30T07:44:59.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.587+0000 7f905f7fe640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f904c073f60 0x7f904c076400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.587+0000 7f905f7fe640 10 monclient: get_auth_request con 0x7f904c073f60 auth_method 0 2024-03-30T07:44:59.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.591+0000 7f905cff9640 10 client.15921.objecter ms_handle_connect 0x7f904c073f60 2024-03-30T07:44:59.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.591+0000 7f905f7fe640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f904c073f60 0x7f904c076400 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.617+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== osd.7 v2:172.21.15.52:6804/35316 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f904c076ad0 con 0x7f904c073f60 2024-03-30T07:44:59.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.617+0000 7f905cff9640 10 client.15921.objecter ms_dispatch 0x7f9060000b90 command_reply(tid 1: 0 ) v1 2024-03-30T07:44:59.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.617+0000 7f905cff9640 10 client.15921.objecter _finish_command 1 = system:0 2024-03-30T07:44:59.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.617+0000 7f905cff9640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 4 2024-03-30T07:44:59.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.617+0000 7f905cff9640 15 client.15921.objecter _session_command_op_remove 7 1 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 10 client.15921.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 15 client.15921.objecter _session_command_op_assign -1 2 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 20 client.15921.objecter _get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 20 client.15921.objecter _get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 15 client.15921.objecter _session_command_op_remove -1 2 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 20 client.15921.objecter get_session s=0x7f904c073890 osd=7 3 2024-03-30T07:44:59.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 15 client.15921.objecter _session_command_op_assign 7 2 2024-03-30T07:44:59.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 4 2024-03-30T07:44:59.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 10 client.15921.objecter _send_command 2 2024-03-30T07:44:59.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.629+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 --> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f90600b3990 con 0x7f904c073f60 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f905cff9640 1 -- 172.21.15.47:0/1464519951 <== osd.7 v2:172.21.15.52:6804/35316 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f90600b3990 con 0x7f904c073f60 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f905cff9640 10 client.15921.objecter ms_dispatch 0x7f9060000b90 command_reply(tid 2: 0 ) v1 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f905cff9640 10 client.15921.objecter _finish_command 2 = system:0 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f905cff9640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 4 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f905cff9640 15 client.15921.objecter _session_command_op_remove 7 2 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T07:44:59.647 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 10 librados: watch_flush enter 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 10 librados: watch_flush exit 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 10 client.15921.objecter close_session for osd.7 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f904c073f60 msgr2=0x7f904c076400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6804/35316,v1:172.21.15.52:6806/35316] conn(0x7f904c073f60 0x7f904c076400 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 20 client.15921.objecter put_session s=0x7f904c073890 osd=7 1 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 20 client.15921.objecter shutdown clearing up homeless session... 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 10 client.15921.objecter successfully canceled tick 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f904c06c590 msgr2=0x7f904c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f904c06c590 0x7f904c06ea50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9050035000 tx=0x7f9050030000 comp rx=0 tx=0).stop 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 10 monclient: shutdown 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 msgr2=0x7f9060131410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 --2- 172.21.15.47:0/1464519951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060141d60 0x7f9060131410 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f905800b970 tx=0x7f905800be30 comp rx=0 tx=0).stop 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f9064e53640 1 -- 172.21.15.47:0/1464519951 reap_dead start 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 shutdown_connections 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 >> 172.21.15.47:0/1464519951 conn(0x7f9060065bf0 msgr2=0x7f90600b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 shutdown_connections 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 -- 172.21.15.47:0/1464519951 wait complete. 2024-03-30T07:44:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.645+0000 7f90668b6640 1 librados: shutdown 2024-03-30T07:44:59.658 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-30T07:44:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.757+0000 7f757bfd3640 10 monclient: get_monmap_and_config 2024-03-30T07:44:59.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.757+0000 7f757bfd3640 10 monclient: build_initial_monmap 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.757+0000 7f757bfd3640 10 monclient: monmap: 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:44:59.758221+0000 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:44:59.758221+0000 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 Processor -- start 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 -- start start 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient: init 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f75740b4280 0x7f75740a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-c con 0x7f75740b4280 addr v2:172.21.15.52:3300/0 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f757413e930 addr v1:172.21.15.52:6789/0 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-b con 0x7f757413eda0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:59.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740aa3c0 con 0x7f757413e930 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75740b4660 con 0x7f75740b4280 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740aa630 con 0x7f757413eda0 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.758+0000 7f757bfd3640 10 monclient(hunting): authenticate will time out at 3225.874087s 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:52748/0 (socket says 172.21.15.47:52748) 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 1 -- 172.21.15.47:0/2905110462 learned_addr learned my addr 172.21.15.47:0/2905110462 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 10 monclient(hunting): get_auth_request con 0x7f757413eda0 auth_method 0 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.759+0000 7f757a570640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.760+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.760+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.760+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.761+0000 7f7579d6f640 1 --2- 172.21.15.47:0/2905110462 >> v2:172.21.15.52:3300/0 conn(0x7f75740b4280 0x7f75740a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.761+0000 7f7579d6f640 10 monclient(hunting): get_auth_request con 0x7f75740b4280 auth_method 0 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.761+0000 7f7579d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.761+0000 7f7579d6f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.761+0000 7f7579d6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 10 monclient(hunting): handle_auth_done global_id 17132 payload 274 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 1 -- 172.21.15.47:0/2905110462 >> v2:172.21.15.52:3300/0 conn(0x7f75740b4280 msgr2=0x7f75740a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 1 --2- 172.21.15.47:0/2905110462 >> v2:172.21.15.52:3300/0 conn(0x7f75740b4280 0x7f75740a8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 1 -- 172.21.15.47:0/2905110462 >> v1:172.21.15.52:6789/0 conn(0x7f757413e930 legacy=0x7f7574141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 10 monclient: _finish_hunting 0 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 1 monclient: found mon.noname-b 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:44:59.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.762+0000 7f757a570640 1 -- 172.21.15.47:0/2905110462 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740a9210 con 0x7f757413eda0 2024-03-30T07:44:59.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.763+0000 7f757a570640 1 --2- 172.21.15.47:0/2905110462 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f756c009e50 tx=0x7f756c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b1e7f5ed23c058 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 1 -- 172.21.15.47:0/2905110462 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c02c9e0 con 0x7f757413eda0 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: dump: 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.771 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: _finish_auth 0 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: _check_auth_tickets 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 1 -- 172.21.15.47:0/2905110462 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c02cb40 con 0x7f757413eda0 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 1 -- 172.21.15.47:0/2905110462 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c0343f0 con 0x7f757413eda0 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:44:59.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f7578d6d640 10 monclient: dump: 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f757bfd3640 5 monclient: authenticate success, global_id 17132 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f757bfd3640 10 monclient: get_monmap_and_config success 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.767+0000 7f757bfd3640 4 set_mon_vals no callback set 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 10 monclient: shutdown 2024-03-30T07:44:59.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 -- 172.21.15.47:0/2905110462 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 msgr2=0x7f7574141d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 --2- 172.21.15.47:0/2905110462 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f756c009e50 tx=0x7f756c02be40 comp rx=0 tx=0).stop 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 -- 172.21.15.47:0/2905110462 shutdown_connections 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 --2- 172.21.15.47:0/2905110462 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f7574141d80 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 --2- 172.21.15.47:0/2905110462 >> v2:172.21.15.52:3300/0 conn(0x7f75740b4280 0x7f75740a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 -- 172.21.15.47:0/2905110462 >> 172.21.15.47:0/2905110462 conn(0x7f7574065e00 msgr2=0x7f7574066210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 -- 172.21.15.47:0/2905110462 shutdown_connections 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 1 -- 172.21.15.47:0/2905110462 wait complete. 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.768+0000 7f757bfd3640 10 monclient: build_initial_monmap 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 10 monclient: monmap: 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:44:59.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 librados: starting msgr at 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 librados: starting objecter 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 Processor -- start 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 -- start start 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 librados: setting wanted keys 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 1 librados: calling monclient init 2024-03-30T07:44:59.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.769+0000 7f757bfd3640 10 monclient: init 2024-03-30T07:44:59.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient: _reopen_session rank -1 2024-03-30T07:44:59.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 0x7f757414ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-b con 0x7f75740b4280 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757413e930 0x7f75741d6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-a con 0x7f757413e930 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f75741d6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): picked mon.noname-c con 0x7f757413eda0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757414a3a0 con 0x7f757413e930 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757414a5a0 con 0x7f75740b4280 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): start opening mon connection 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75741d6330 con 0x7f757413eda0 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): _renew_subs 2024-03-30T07:44:59.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757bfd3640 10 monclient(hunting): authenticate will time out at 3225.885235s 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f75741d6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757956e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757413e930 0x7f75741d6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f7579d6f640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 0x7f757414ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f75741d6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:52758/0 (socket says 172.21.15.47:52758) 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 1 -- 172.21.15.47:0/205339417 learned_addr learned my addr 172.21.15.47:0/205339417 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 10 monclient(hunting): get_auth_request con 0x7f757413eda0 auth_method 0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757a570640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757956e640 10 monclient(hunting): get_auth_request con 0x7f757413e930 auth_method 0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757956e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757956e640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.770+0000 7f757956e640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): get_auth_request con 0x7f75740b4280 auth_method 0 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:44:59.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f7579d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757a570640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757956e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757956e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.771+0000 7f757956e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 10 monclient(hunting): handle_auth_done global_id 22420 payload 824 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 -- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 msgr2=0x7f75741d6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f75741d6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 -- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757413e930 msgr2=0x7f75741d6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757413e930 0x7f75741d6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 10 monclient: _finish_hunting 0 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 monclient: found mon.noname-b 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:44:59.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.774+0000 7f7579d6f640 1 -- 172.21.15.47:0/205339417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c009b30 con 0x7f75740b4280 2024-03-30T07:44:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.776+0000 7f7579d6f640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 0x7f757414ad30 secure :-1 s=READY pgs=5530 cs=0 l=1 rev1=1 crypto rx=0x7f757000d5f0 tx=0x7f757000dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:44:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.776+0000 7f756a7fc640 10 client.?.objecter ms_handle_connect 0x7f75740b4280 2024-03-30T07:44:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.776+0000 7f756a7fc640 10 client.?.objecter resend_mon_ops 2024-03-30T07:44:59.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75700125f0 con 0x7f75740b4280 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: dump: 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.783 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: _finish_auth 0 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7570012750 con 0x7f75740b4280 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7570012b00 con 0x7f75740b4280 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 5 monclient: authenticate success, global_id 22420 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756a7fc640 10 monclient: dump: 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:44:59.784 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f756bfff640 4 set_mon_vals no callback set 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 10 monclient: _renew_subs 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574147ed0 con 0x7f75740b4280 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 10 client.22420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 10 monclient: _renew_subs 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:59.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75741484e0 con 0x7f75740b4280 2024-03-30T07:44:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.780+0000 7f757bfd3640 1 librados: init done 2024-03-30T07:44:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.781+0000 7f757bfd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.781+0000 7f757bfd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:44:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.781+0000 7f757bfd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:44:59.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.781+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75741d64f0 con 0x7f75740b4280 2024-03-30T07:44:59.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.800+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7570012c60 con 0x7f75740b4280 2024-03-30T07:44:59.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.801+0000 7f756a7fc640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 0x7f755806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:44:59.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.801+0000 7f757956e640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 0x7f755806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:44:59.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.801+0000 7f757956e640 10 monclient: get_auth_request con 0x7f755806c610 auth_method 0 2024-03-30T07:44:59.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.801+0000 7f756a7fc640 10 client.22420.objecter ms_handle_connect 0x7f755806c610 2024-03-30T07:44:59.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:44:59.801+0000 7f757956e640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 0x7f755806ead0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f757413d680 tx=0x7f7564009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:00.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.051+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f75700590c0 con 0x7f75740b4280 2024-03-30T07:45:00.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.051+0000 7f756a7fc640 10 client.22420.objecter ms_dispatch 0x7f7574000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:45:00.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.051+0000 7f756a7fc640 3 client.22420.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:45:00.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.051+0000 7f756a7fc640 3 client.22420.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:45:00.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.052+0000 7f756a7fc640 20 client.22420.objecter dump_active .. 0 homeless 2024-03-30T07:45:00.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.063+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f7570055400 con 0x7f75740b4280 2024-03-30T07:45:00.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.063+0000 7f756a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:00.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.063+0000 7f756a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:45:00.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.249+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f75740a8ed0 con 0x7f755806c610 2024-03-30T07:45:00.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.250+0000 7f756a7fc640 1 -- 172.21.15.47:0/205339417 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7564001440 con 0x7f755806c610 2024-03-30T07:45:00.252 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 7 to deep-scrub 2024-03-30T07:45:00.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.251+0000 7f757bfd3640 10 librados: watch_flush enter 2024-03-30T07:45:00.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 10 librados: watch_flush exit 2024-03-30T07:45:00.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 20 client.22420.objecter shutdown clearing up homeless session... 2024-03-30T07:45:00.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 10 client.22420.objecter successfully canceled tick 2024-03-30T07:45:00.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 msgr2=0x7f755806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 0x7f755806ead0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f757413d680 tx=0x7f7564009040 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 10 monclient: shutdown 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 msgr2=0x7f757414ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 0x7f757414ad30 secure :-1 s=READY pgs=5530 cs=0 l=1 rev1=1 crypto rx=0x7f757000d5f0 tx=0x7f757000dab0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 shutdown_connections 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f755806c610 0x7f755806ead0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f757413eda0 0x7f75741d6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f757413e930 0x7f75741d6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 --2- 172.21.15.47:0/205339417 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f75740b4280 0x7f757414ad30 unknown :-1 s=CLOSED pgs=5530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 >> 172.21.15.47:0/205339417 conn(0x7f7574065e00 msgr2=0x7f7574141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 shutdown_connections 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 -- 172.21.15.47:0/205339417 wait complete. 2024-03-30T07:45:00.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.252+0000 7f757bfd3640 1 librados: shutdown 2024-03-30T07:45:00.265 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:45:00.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.365+0000 7f96c642d640 10 monclient: get_monmap_and_config 2024-03-30T07:45:00.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.365+0000 7f96c642d640 10 monclient: build_initial_monmap 2024-03-30T07:45:00.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.365+0000 7f96c642d640 10 monclient: monmap: 2024-03-30T07:45:00.367 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:45:00.366143+0000 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:45:00.366143+0000 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:45:00.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 Processor -- start 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 -- start start 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient: init 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f96c0073680 0x7f96c013e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-c con 0x7f96c0073680 addr v2:172.21.15.52:3300/0 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96c00af060 addr v1:172.21.15.47:6789/0 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 0x7f96c013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-b con 0x7f96c00af760 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c006e680 con 0x7f96c00af060 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c0073a60 con 0x7f96c0073680 2024-03-30T07:45:00.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00afc20 con 0x7f96c00af760 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.366+0000 7f96c642d640 10 monclient(hunting): authenticate will time out at 3226.481861s 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96c49ca640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 0x7f96c013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bf7fe640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f96c00af060 0x7f96c0144b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33250/0 (socket says 172.21.15.47:33250) 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bf7fe640 1 -- 172.21.15.47:0/1276039913 learned_addr learned my addr 172.21.15.47:0/1276039913 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bffff640 1 --2- 172.21.15.47:0/1276039913 >> v2:172.21.15.52:3300/0 conn(0x7f96c0073680 0x7f96c013e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96c49ca640 10 monclient(hunting): get_auth_request con 0x7f96c00af760 auth_method 0 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96c49ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96c49ca640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96c49ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bffff640 10 monclient(hunting): get_auth_request con 0x7f96c0073680 auth_method 0 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bffff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:00.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.367+0000 7f96bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:00.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.368+0000 7f96bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 10 monclient(hunting): handle_auth_done global_id 17138 payload 274 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 1 -- 172.21.15.47:0/1276039913 >> v2:172.21.15.52:3300/0 conn(0x7f96c0073680 msgr2=0x7f96c013e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 1 --2- 172.21.15.47:0/1276039913 >> v2:172.21.15.52:3300/0 conn(0x7f96c0073680 0x7f96c013e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 1 -- 172.21.15.47:0/1276039913 >> v1:172.21.15.47:6789/0 conn(0x7f96c00af060 legacy=0x7f96c0144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 10 monclient: _finish_hunting 0 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 1 monclient: found mon.noname-b 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.369+0000 7f96c49ca640 1 -- 172.21.15.47:0/1276039913 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c0145240 con 0x7f96c00af760 2024-03-30T07:45:00.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.371+0000 7f96c49ca640 1 --2- 172.21.15.47:0/1276039913 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 0x7f96c013ed90 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009d90 tx=0x7f96b802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a0702c6fc73f50e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 1 -- 172.21.15.47:0/1276039913 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b802cbc0 con 0x7f96c00af760 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: dump: 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:00.379 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: _finish_auth 0 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: _check_auth_tickets 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 1 -- 172.21.15.47:0/1276039913 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b802cd20 con 0x7f96c00af760 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 5 monclient: authenticate success, global_id 17138 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 10 monclient: get_monmap_and_config success 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 1 -- 172.21.15.47:0/1276039913 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8034660 con 0x7f96c00af760 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 4 set_mon_vals no callback set 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:00.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96beffd640 10 monclient: dump: 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 10 monclient: shutdown 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:00.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 1 -- 172.21.15.47:0/1276039913 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 msgr2=0x7f96c013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 1 --2- 172.21.15.47:0/1276039913 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 0x7f96c013ed90 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009d90 tx=0x7f96b802be10 comp rx=0 tx=0).stop 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.376+0000 7f96c642d640 1 -- 172.21.15.47:0/1276039913 shutdown_connections 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 1 --2- 172.21.15.47:0/1276039913 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af760 0x7f96c013ed90 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 1 --2- 172.21.15.47:0/1276039913 >> v2:172.21.15.52:3300/0 conn(0x7f96c0073680 0x7f96c013e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 1 -- 172.21.15.47:0/1276039913 >> 172.21.15.47:0/1276039913 conn(0x7f96c0069ec0 msgr2=0x7f96c006a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 1 -- 172.21.15.47:0/1276039913 shutdown_connections 2024-03-30T07:45:00.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 1 -- 172.21.15.47:0/1276039913 wait complete. 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 10 monclient: build_initial_monmap 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.377+0000 7f96c642d640 10 monclient: monmap: 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:45:00.385 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 librados: starting msgr at 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 librados: starting objecter 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 Processor -- start 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 -- start start 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 librados: setting wanted keys 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 1 librados: calling monclient init 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.378+0000 7f96c642d640 10 monclient: init 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96c0073680 0x7f96c013f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-a con 0x7f96c0073680 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af060 0x7f96c013d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-c con 0x7f96c00af060 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:00.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): picked mon.noname-b con 0x7f96c00af760 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01d6880 con 0x7f96c0073680 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c013d840 con 0x7f96c00af760 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c013c9d0 con 0x7f96c00af060 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c642d640 10 monclient(hunting): authenticate will time out at 3226.494277s 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.379+0000 7f96c49ca640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.52:3300/0 says I am v2:172.21.15.47:52260/0 (socket says 172.21.15.47:52260) 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 1 -- 172.21.15.47:0/2233455902 learned_addr learned my addr 172.21.15.47:0/2233455902 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): get_auth_request con 0x7f96c00af760 auth_method 0 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:00.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96bf7fe640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af060 0x7f96c013d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.380+0000 7f96c49ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): get_auth_request con 0x7f96c00af060 auth_method 0 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.381+0000 7f96bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 10 monclient(hunting): handle_auth_done global_id 22426 payload 824 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 -- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af060 msgr2=0x7f96c013d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af060 0x7f96c013d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 -- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96c0073680 msgr2=0x7f96c013f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96c0073680 0x7f96c013f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 10 monclient: _finish_hunting 0 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 monclient: found mon.noname-b 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.52:3300/0 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.383+0000 7f96c49ca640 1 -- 172.21.15.47:0/2233455902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8009a70 con 0x7f96c00af760 2024-03-30T07:45:00.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.385+0000 7f96c49ca640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 secure :-1 s=READY pgs=5531 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009eb0 tx=0x7f96b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:00.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.385+0000 7f96bcff9640 10 client.?.objecter ms_handle_connect 0x7f96c00af760 2024-03-30T07:45:00.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.385+0000 7f96bcff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b803e610 con 0x7f96c00af760 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: dump: 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.392 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: _finish_auth 0 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: _check_auth_tickets 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b802ce90 con 0x7f96c00af760 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 5 monclient: authenticate success, global_id 22426 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 10 monclient: _renew_subs 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:45:00.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c013cbf0 con 0x7f96c00af760 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b803e9b0 con 0x7f96c00af760 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96bcff9640 10 monclient: dump: 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:00.394 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96be7fc640 4 set_mon_vals no callback set 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 10 client.22426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 10 monclient: _renew_subs 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c01410a0 con 0x7f96c00af760 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.389+0000 7f96c642d640 1 librados: init done 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.390+0000 7f96c642d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.390+0000 7f96c642d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.390+0000 7f96c642d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.52:3300/0 2024-03-30T07:45:00.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.390+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c004e030 con 0x7f96c00af760 2024-03-30T07:45:00.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.409+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f96b803b050 con 0x7f96c00af760 2024-03-30T07:45:00.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.410+0000 7f96bcff9640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 0x7f96ac06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:00.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.410+0000 7f96bffff640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 0x7f96ac06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:00.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.410+0000 7f96bffff640 10 monclient: get_auth_request con 0x7f96ac06c5e0 auth_method 0 2024-03-30T07:45:00.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.410+0000 7f96bcff9640 10 client.22426.objecter ms_handle_connect 0x7f96ac06c5e0 2024-03-30T07:45:00.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.410+0000 7f96bffff640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 0x7f96ac06eaa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f96b0006a90 tx=0x7f96b0006a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:00.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.414+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f96b807ad80 con 0x7f96c00af760 2024-03-30T07:45:00.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.414+0000 7f96bcff9640 10 client.22426.objecter ms_dispatch 0x7f96c0000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:45:00.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.414+0000 7f96bcff9640 3 client.22426.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:45:00.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.414+0000 7f96bcff9640 3 client.22426.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:45:00.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.414+0000 7f96bcff9640 20 client.22426.objecter dump_active .. 0 homeless 2024-03-30T07:45:00.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.708+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mon.1 v2:172.21.15.52:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f96b80770c0 con 0x7f96c00af760 2024-03-30T07:45:00.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.708+0000 7f96bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:00.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.708+0000 7f96bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:45:00.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.894+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96c013f0b0 con 0x7f96ac06c5e0 2024-03-30T07:45:00.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.899+0000 7f96bcff9640 1 -- 172.21.15.47:0/2233455902 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151296 (secure 0 0 0) 0x7f96c013f0b0 con 0x7f96ac06c5e0 2024-03-30T07:45:00.902 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:45:00.903 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 10 librados: watch_flush enter 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 10 librados: watch_flush exit 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 20 client.22426.objecter shutdown clearing up homeless session... 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 10 client.22426.objecter successfully canceled tick 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 msgr2=0x7f96ac06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 0x7f96ac06eaa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f96b0006a90 tx=0x7f96b0006a20 comp rx=0 tx=0).stop 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 10 monclient: shutdown 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 msgr2=0x7f96c01d6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 secure :-1 s=READY pgs=5531 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009eb0 tx=0x7f96b8004a40 comp rx=0 tx=0).stop 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 shutdown_connections 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f96ac06c5e0 0x7f96ac06eaa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f96c00af760 0x7f96c01d6da0 unknown :-1 s=CLOSED pgs=5531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96c00af060 0x7f96c013d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 --2- 172.21.15.47:0/2233455902 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96c0073680 0x7f96c013f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 >> 172.21.15.47:0/2233455902 conn(0x7f96c0069ec0 msgr2=0x7f96c013dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 shutdown_connections 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 -- 172.21.15.47:0/2233455902 wait complete. 2024-03-30T07:45:00.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:00.901+0000 7f96c642d640 1 librados: shutdown 2024-03-30T07:45:00.914 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1853,"stamp":"2024-03-30T07:45:00.373643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529580,"kb_used_data":59192,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445140,"statfs":{"total":773094113280,"available":772551823360,"internally_reserved":0,"allocated":60612608,"data_stored":34024346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.164295"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":934,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:54.307655+0000","last_change":"2024-03-30T07:18:12.759583+0000","last_active":"2024-03-30T07:44:54.307655+0000","last_peered":"2024-03-30T07:44:54.307655+0000","last_clean":"2024-03-30T07:44:54.307655+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:44:54.307655+0000","last_undegraded":"2024-03-30T07:44:54.307655+0000","last_fullsized":"2024-03-30T07:44:54.307655+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":947,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:54.310451+0000","last_change":"2024-03-30T07:24:14.842838+0000","last_active":"2024-03-30T07:44:54.310451+0000","last_peered":"2024-03-30T07:44:54.310451+0000","last_clean":"2024-03-30T07:44:54.310451+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:44:54.310451+0000","last_undegraded":"2024-03-30T07:44:54.310451+0000","last_fullsized":"2024-03-30T07:44:54.310451+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":920,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.161769+0000","last_change":"2024-03-30T07:18:08.005915+0000","last_active":"2024-03-30T07:44:48.161769+0000","last_peered":"2024-03-30T07:44:48.161769+0000","last_clean":"2024-03-30T07:44:48.161769+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:44:48.161769+0000","last_undegraded":"2024-03-30T07:44:48.161769+0000","last_fullsized":"2024-03-30T07:44:48.161769+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:57:08.402927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":933,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.293264+0000","last_change":"2024-03-30T07:24:15.826961+0000","last_active":"2024-03-30T07:44:49.293264+0000","last_peered":"2024-03-30T07:44:49.293264+0000","last_clean":"2024-03-30T07:44:49.293264+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:44:49.293264+0000","last_undegraded":"2024-03-30T07:44:49.293264+0000","last_fullsized":"2024-03-30T07:44:49.293264+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":913,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.649486+0000","last_change":"2024-03-30T07:18:07.657409+0000","last_active":"2024-03-30T07:44:49.649486+0000","last_peered":"2024-03-30T07:44:49.649486+0000","last_clean":"2024-03-30T07:44:49.649486+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:44:49.649486+0000","last_undegraded":"2024-03-30T07:44:49.649486+0000","last_fullsized":"2024-03-30T07:44:49.649486+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:39:31.459285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":924,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.658684+0000","last_change":"2024-03-30T07:44:18.246797+0000","last_active":"2024-03-30T07:44:49.658684+0000","last_peered":"2024-03-30T07:44:49.658684+0000","last_clean":"2024-03-30T07:44:49.658684+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:44:49.658684+0000","last_undegraded":"2024-03-30T07:44:49.658684+0000","last_fullsized":"2024-03-30T07:44:49.658684+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:11.652670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":937,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.354347+0000","last_change":"2024-03-30T07:44:23.336397+0000","last_active":"2024-03-30T07:44:48.354347+0000","last_peered":"2024-03-30T07:44:48.354347+0000","last_clean":"2024-03-30T07:44:48.354347+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:44:48.354347+0000","last_undegraded":"2024-03-30T07:44:48.354347+0000","last_fullsized":"2024-03-30T07:44:48.354347+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:03:38.840636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.236285+0000","last_change":"2024-03-30T07:17:58.934819+0000","last_active":"2024-03-30T07:44:49.236285+0000","last_peered":"2024-03-30T07:44:49.236285+0000","last_clean":"2024-03-30T07:44:49.236285+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:44:49.236285+0000","last_undegraded":"2024-03-30T07:44:49.236285+0000","last_fullsized":"2024-03-30T07:44:49.236285+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:40:48.021215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":925,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.664652+0000","last_change":"2024-03-30T07:24:15.696366+0000","last_active":"2024-03-30T07:44:49.664652+0000","last_peered":"2024-03-30T07:44:49.664652+0000","last_clean":"2024-03-30T07:44:49.664652+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:44:49.664652+0000","last_undegraded":"2024-03-30T07:44:49.664652+0000","last_fullsized":"2024-03-30T07:44:49.664652+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:36:53.871311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":911,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.675934+0000","last_change":"2024-03-30T07:18:07.825982+0000","last_active":"2024-03-30T07:44:49.675934+0000","last_peered":"2024-03-30T07:44:49.675934+0000","last_clean":"2024-03-30T07:44:49.675934+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:49.675934+0000","last_undegraded":"2024-03-30T07:44:49.675934+0000","last_fullsized":"2024-03-30T07:44:49.675934+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:07:32.853655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":927,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.677765+0000","last_change":"2024-03-30T07:24:13.737991+0000","last_active":"2024-03-30T07:44:49.677765+0000","last_peered":"2024-03-30T07:44:49.677765+0000","last_clean":"2024-03-30T07:44:49.677765+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:44:49.677765+0000","last_undegraded":"2024-03-30T07:44:49.677765+0000","last_fullsized":"2024-03-30T07:44:49.677765+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":911,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.684355+0000","last_change":"2024-03-30T07:18:16.996109+0000","last_active":"2024-03-30T07:44:49.684355+0000","last_peered":"2024-03-30T07:44:49.684355+0000","last_clean":"2024-03-30T07:44:49.684355+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:44:49.684355+0000","last_undegraded":"2024-03-30T07:44:49.684355+0000","last_fullsized":"2024-03-30T07:44:49.684355+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:46:51.546133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":921,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:56.850388+0000","last_change":"2024-03-30T07:16:49.937668+0000","last_active":"2024-03-30T07:44:56.850388+0000","last_peered":"2024-03-30T07:44:56.850388+0000","last_clean":"2024-03-30T07:44:56.850388+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:44:56.850388+0000","last_undegraded":"2024-03-30T07:44:56.850388+0000","last_fullsized":"2024-03-30T07:44:56.850388+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":920,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.155459+0000","last_change":"2024-03-30T07:18:21.573712+0000","last_active":"2024-03-30T07:44:48.155459+0000","last_peered":"2024-03-30T07:44:48.155459+0000","last_clean":"2024-03-30T07:44:48.155459+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:44:48.155459+0000","last_undegraded":"2024-03-30T07:44:48.155459+0000","last_fullsized":"2024-03-30T07:44:48.155459+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:18:39.319613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":917,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.679783+0000","last_change":"2024-03-30T07:16:51.136640+0000","last_active":"2024-03-30T07:44:49.679783+0000","last_peered":"2024-03-30T07:44:49.679783+0000","last_clean":"2024-03-30T07:44:49.679783+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:49.679783+0000","last_undegraded":"2024-03-30T07:44:49.679783+0000","last_fullsized":"2024-03-30T07:44:49.679783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:42:45.877986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":911,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.666273+0000","last_change":"2024-03-30T07:18:16.070084+0000","last_active":"2024-03-30T07:44:49.666273+0000","last_peered":"2024-03-30T07:44:49.666273+0000","last_clean":"2024-03-30T07:44:49.666273+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:44:49.666273+0000","last_undegraded":"2024-03-30T07:44:49.666273+0000","last_fullsized":"2024-03-30T07:44:49.666273+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:20:22.115137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":921,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.674736+0000","last_change":"2024-03-30T07:16:49.703176+0000","last_active":"2024-03-30T07:44:49.674736+0000","last_peered":"2024-03-30T07:44:49.674736+0000","last_clean":"2024-03-30T07:44:49.674736+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:44:49.674736+0000","last_undegraded":"2024-03-30T07:44:49.674736+0000","last_fullsized":"2024-03-30T07:44:49.674736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":910,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.682236+0000","last_change":"2024-03-30T07:18:17.007226+0000","last_active":"2024-03-30T07:44:49.682236+0000","last_peered":"2024-03-30T07:44:49.682236+0000","last_clean":"2024-03-30T07:44:49.682236+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:44:49.682236+0000","last_undegraded":"2024-03-30T07:44:49.682236+0000","last_fullsized":"2024-03-30T07:44:49.682236+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:02:56.775071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":954,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:58.572543+0000","last_change":"2024-03-30T07:44:18.307851+0000","last_active":"2024-03-30T07:44:58.572543+0000","last_peered":"2024-03-30T07:44:58.572543+0000","last_clean":"2024-03-30T07:44:58.572543+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:44:58.572543+0000","last_undegraded":"2024-03-30T07:44:58.572543+0000","last_fullsized":"2024-03-30T07:44:58.572543+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":920,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.164260+0000","last_change":"2024-03-30T07:17:58.343146+0000","last_active":"2024-03-30T07:44:48.164260+0000","last_peered":"2024-03-30T07:44:48.164260+0000","last_clean":"2024-03-30T07:44:48.164260+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:44:48.164260+0000","last_undegraded":"2024-03-30T07:44:48.164260+0000","last_fullsized":"2024-03-30T07:44:48.164260+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:24:03.491281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":935,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.665989+0000","last_change":"2024-03-30T07:44:23.549306+0000","last_active":"2024-03-30T07:44:49.665989+0000","last_peered":"2024-03-30T07:44:49.665989+0000","last_clean":"2024-03-30T07:44:49.665989+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:44:49.665989+0000","last_undegraded":"2024-03-30T07:44:49.665989+0000","last_fullsized":"2024-03-30T07:44:49.665989+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":928,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.333416+0000","last_change":"2024-03-30T07:44:18.161552+0000","last_active":"2024-03-30T07:44:48.333416+0000","last_peered":"2024-03-30T07:44:48.333416+0000","last_clean":"2024-03-30T07:44:48.333416+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:44:48.333416+0000","last_undegraded":"2024-03-30T07:44:48.333416+0000","last_fullsized":"2024-03-30T07:44:48.333416+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:18.359722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":944,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.142239+0000","last_change":"2024-03-30T07:24:14.186380+0000","last_active":"2024-03-30T07:44:48.142239+0000","last_peered":"2024-03-30T07:44:48.142239+0000","last_clean":"2024-03-30T07:44:48.142239+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:44:48.142239+0000","last_undegraded":"2024-03-30T07:44:48.142239+0000","last_fullsized":"2024-03-30T07:44:48.142239+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:58:59.484353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":922,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.144777+0000","last_change":"2024-03-30T07:18:07.807081+0000","last_active":"2024-03-30T07:44:48.144777+0000","last_peered":"2024-03-30T07:44:48.144777+0000","last_clean":"2024-03-30T07:44:48.144777+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:44:48.144777+0000","last_undegraded":"2024-03-30T07:44:48.144777+0000","last_fullsized":"2024-03-30T07:44:48.144777+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:35:37.547969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.658595+0000","last_change":"2024-03-30T07:24:13.390417+0000","last_active":"2024-03-30T07:44:49.658595+0000","last_peered":"2024-03-30T07:44:49.658595+0000","last_clean":"2024-03-30T07:44:49.658595+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:49.658595+0000","last_undegraded":"2024-03-30T07:44:49.658595+0000","last_fullsized":"2024-03-30T07:44:49.658595+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":946,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:50.952155+0000","last_change":"2024-03-30T07:18:17.831393+0000","last_active":"2024-03-30T07:44:50.952155+0000","last_peered":"2024-03-30T07:44:50.952155+0000","last_clean":"2024-03-30T07:44:50.952155+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:50.952155+0000","last_undegraded":"2024-03-30T07:44:50.952155+0000","last_fullsized":"2024-03-30T07:44:50.952155+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":921,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.671170+0000","last_change":"2024-03-30T07:16:49.957420+0000","last_active":"2024-03-30T07:44:49.671170+0000","last_peered":"2024-03-30T07:44:49.671170+0000","last_clean":"2024-03-30T07:44:49.671170+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:49.671170+0000","last_undegraded":"2024-03-30T07:44:49.671170+0000","last_fullsized":"2024-03-30T07:44:49.671170+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:16:48.116623+0000","last_clean_scrub_stamp":"2024-03-30T07:16:48.116623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"994'78","reported_seq":1091,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:56.280200+0000","last_change":"2024-03-30T07:44:56.282541+0000","last_active":"2024-03-30T07:44:56.280200+0000","last_peered":"2024-03-30T07:44:56.280200+0000","last_clean":"2024-03-30T07:44:56.280200+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:56.280200+0000","last_undegraded":"2024-03-30T07:44:56.280200+0000","last_fullsized":"2024-03-30T07:44:56.280200+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"994'78","last_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_deep_scrub":"994'78","last_deep_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_clean_scrub_stamp":"2024-03-30T07:44:56.279300+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:20:04.428365+0000","scrub_duration":0.11938407600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":951,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:50.948735+0000","last_change":"2024-03-30T07:18:08.767217+0000","last_active":"2024-03-30T07:44:50.948735+0000","last_peered":"2024-03-30T07:44:50.948735+0000","last_clean":"2024-03-30T07:44:50.948735+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:44:50.948735+0000","last_undegraded":"2024-03-30T07:44:50.948735+0000","last_fullsized":"2024-03-30T07:44:50.948735+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":958,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:50.958764+0000","last_change":"2024-03-30T07:24:13.784512+0000","last_active":"2024-03-30T07:44:50.958764+0000","last_peered":"2024-03-30T07:44:50.958764+0000","last_clean":"2024-03-30T07:44:50.958764+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:44:50.958764+0000","last_undegraded":"2024-03-30T07:44:50.958764+0000","last_fullsized":"2024-03-30T07:44:50.958764+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":927,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.355991+0000","last_change":"2024-03-30T07:44:18.218063+0000","last_active":"2024-03-30T07:44:48.355991+0000","last_peered":"2024-03-30T07:44:48.355991+0000","last_clean":"2024-03-30T07:44:48.355991+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:44:48.355991+0000","last_undegraded":"2024-03-30T07:44:48.355991+0000","last_fullsized":"2024-03-30T07:44:48.355991+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:41:46.659263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":914,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:52.454172+0000","last_change":"2024-03-30T07:18:04.168467+0000","last_active":"2024-03-30T07:44:52.454172+0000","last_peered":"2024-03-30T07:44:52.454172+0000","last_clean":"2024-03-30T07:44:52.454172+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:44:52.454172+0000","last_undegraded":"2024-03-30T07:44:52.454172+0000","last_fullsized":"2024-03-30T07:44:52.454172+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":936,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.134456+0000","last_change":"2024-03-30T07:24:14.248616+0000","last_active":"2024-03-30T07:44:48.134456+0000","last_peered":"2024-03-30T07:44:48.134456+0000","last_clean":"2024-03-30T07:44:48.134456+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:44:48.134456+0000","last_undegraded":"2024-03-30T07:44:48.134456+0000","last_fullsized":"2024-03-30T07:44:48.134456+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:23:34.001344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":934,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:54.312608+0000","last_change":"2024-03-30T07:18:53.674429+0000","last_active":"2024-03-30T07:44:54.312608+0000","last_peered":"2024-03-30T07:44:54.312608+0000","last_clean":"2024-03-30T07:44:54.312608+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:44:54.312608+0000","last_undegraded":"2024-03-30T07:44:54.312608+0000","last_fullsized":"2024-03-30T07:44:54.312608+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":953,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:50.961317+0000","last_change":"2024-03-30T07:18:37.331986+0000","last_active":"2024-03-30T07:44:50.961317+0000","last_peered":"2024-03-30T07:44:50.961317+0000","last_clean":"2024-03-30T07:44:50.961317+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:44:50.961317+0000","last_undegraded":"2024-03-30T07:44:50.961317+0000","last_fullsized":"2024-03-30T07:44:50.961317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":909,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.677525+0000","last_change":"2024-03-30T07:18:16.744594+0000","last_active":"2024-03-30T07:44:49.677525+0000","last_peered":"2024-03-30T07:44:49.677525+0000","last_clean":"2024-03-30T07:44:49.677525+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:44:49.677525+0000","last_undegraded":"2024-03-30T07:44:49.677525+0000","last_fullsized":"2024-03-30T07:44:49.677525+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:35:24.403172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":910,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.672207+0000","last_change":"2024-03-30T07:18:16.066842+0000","last_active":"2024-03-30T07:44:49.672207+0000","last_peered":"2024-03-30T07:44:49.672207+0000","last_clean":"2024-03-30T07:44:49.672207+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:44:49.672207+0000","last_undegraded":"2024-03-30T07:44:49.672207+0000","last_fullsized":"2024-03-30T07:44:49.672207+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:16:21.059197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":915,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.669547+0000","last_change":"2024-03-30T07:18:03.197722+0000","last_active":"2024-03-30T07:44:49.669547+0000","last_peered":"2024-03-30T07:44:49.669547+0000","last_clean":"2024-03-30T07:44:49.669547+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:44:49.669547+0000","last_undegraded":"2024-03-30T07:44:49.669547+0000","last_fullsized":"2024-03-30T07:44:49.669547+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":948,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:54.303182+0000","last_change":"2024-03-30T07:44:18.305103+0000","last_active":"2024-03-30T07:44:54.303182+0000","last_peered":"2024-03-30T07:44:54.303182+0000","last_clean":"2024-03-30T07:44:54.303182+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:44:54.303182+0000","last_undegraded":"2024-03-30T07:44:54.303182+0000","last_fullsized":"2024-03-30T07:44:54.303182+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"43'7","reported_seq":942,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-03-30T07:44:48.343259+0000","last_change":"2024-03-30T07:44:18.289552+0000","last_active":"2024-03-30T07:44:48.343259+0000","last_peered":"2024-03-30T07:44:48.343259+0000","last_clean":"2024-03-30T07:44:48.343259+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:44:48.343259+0000","last_undegraded":"2024-03-30T07:44:48.343259+0000","last_fullsized":"2024-03-30T07:44:48.343259+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:06:03.894404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":937,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.248798+0000","last_change":"2024-03-30T07:24:15.802335+0000","last_active":"2024-03-30T07:44:49.248798+0000","last_peered":"2024-03-30T07:44:49.248798+0000","last_clean":"2024-03-30T07:44:49.248798+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:44:49.248798+0000","last_undegraded":"2024-03-30T07:44:49.248798+0000","last_fullsized":"2024-03-30T07:44:49.248798+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280858,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64656,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307184,"statfs":{"total":96636764160,"available":96570556416,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:25 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":18.387,"15min":18.98},"min":{"1min":5.0540000000000003,"5min":4.7649999999999997,"15min":4.7649999999999997},"max":{"1min":12.066000000000001,"5min":249.63200000000001,"15min":292.04199999999997},"last":6.5350000000000001},{"interface":"front","average":{"1min":12.994,"5min":21.666,"15min":20.100000000000001},"min":{"1min":6.0720000000000001,"5min":5.0730000000000004,"15min":4.79},"max":{"1min":39.451999999999998,"5min":248.767,"15min":299.35000000000002},"last":8.4060000000000006}]},{"osd":1,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":11.307,"5min":13.699,"15min":17.414000000000001},"min":{"1min":6.0860000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":24.446999999999999,"5min":104.782,"15min":169.17099999999999},"last":13.959},{"interface":"front","average":{"1min":16.443999999999999,"5min":19.584,"15min":24.741},"min":{"1min":7.851,"5min":7.851,"15min":6.7789999999999999},"max":{"1min":60.030999999999999,"5min":181.46199999999999,"15min":286.52199999999999},"last":7.851}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1180.998,"5min":530.18200000000002,"15min":228.39699999999999},"min":{"1min":9.0640000000000001,"5min":9.0640000000000001,"15min":8.0619999999999994},"max":{"1min":14020.602000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":7.2729999999999997},{"interface":"front","average":{"1min":1157.28,"5min":537.84500000000003,"15min":233.36799999999999},"min":{"1min":12.426,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13714.877,"5min":13714.877,"15min":13714.877},"last":13.601000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":11.715999999999999,"5min":335.99599999999998,"15min":275.75700000000001},"min":{"1min":7.665,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":16.957999999999998,"5min":12556.546,"15min":13820.906000000001},"last":13.021000000000001},{"interface":"front","average":{"1min":14.755000000000001,"5min":341.73700000000002,"15min":281.92700000000002},"min":{"1min":9.9740000000000002,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":18.872,"5min":12557.409,"15min":13821.846},"last":15.938000000000001}]},{"osd":5,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":27.942,"5min":27.231999999999999,"15min":51.024000000000001},"min":{"1min":9.3350000000000009,"5min":9.3350000000000009,"15min":9.3350000000000009},"max":{"1min":161.35400000000001,"5min":254.04300000000001,"15min":4768.4960000000001},"last":14.56},{"interface":"front","average":{"1min":32.487000000000002,"5min":32.831000000000003,"15min":51.594999999999999},"min":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":7.8760000000000003},"max":{"1min":165.96600000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":16.609000000000002}]},{"osd":6,"last update":"Sat Mar 30 07:42:20 2024","interfaces":[{"interface":"back","average":{"1min":57.975000000000001,"5min":40.822000000000003,"15min":28.888000000000002},"min":{"1min":14.178000000000001,"5min":12.103,"15min":6.9530000000000003},"max":{"1min":328.01600000000002,"5min":328.01600000000002,"15min":350.94400000000002},"last":18.638000000000002},{"interface":"front","average":{"1min":32.357999999999997,"5min":32.185000000000002,"15min":29.07},"min":{"1min":9.5039999999999996,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":98.340999999999994,"5min":294.49700000000001,"15min":323.77999999999997},"last":19.509}]},{"osd":7,"last update":"Sat Mar 30 07:44:07 2024","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":21.923999999999999,"15min":337.53199999999998},"min":{"1min":6.7649999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":19.643999999999998,"5min":334.75400000000002,"15min":19293.749},"last":20.460000000000001},{"interface":"front","average":{"1min":14.932,"5min":27.75,"15min":339.43299999999999},"min":{"1min":10.574999999999999,"5min":9.1430000000000007,"15min":8.2799999999999994},"max":{"1min":20.895,"5min":318.11200000000002,"15min":19292.978999999999},"last":17.195}]}]},{"osd":0,"up_from":22,"seq":94489280859,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60564,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311276,"statfs":{"total":96636764160,"available":96574746624,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":10.212999999999999},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":10.473000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":9.5999999999999996},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":13.611000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":1499.876,"5min":472.52499999999998,"15min":195.184},"min":{"1min":9.7899999999999991,"5min":9.7899999999999991,"15min":6.6429999999999998},"max":{"1min":13546.239,"5min":13546.239,"15min":13546.239},"last":14.843999999999999},{"interface":"front","average":{"1min":1557.163,"5min":474.86099999999999,"15min":194.38},"min":{"1min":7.4000000000000004,"5min":6.7530000000000001,"15min":5.2569999999999997},"max":{"1min":13851.101000000001,"5min":13851.101000000001,"15min":13851.101000000001},"last":13.003}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":14.249000000000001},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":11.598000000000001}]},{"osd":5,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":41.381999999999998,"5min":25.181000000000001,"15min":60.106000000000002},"min":{"1min":7.4539999999999997,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":460.38799999999998,"5min":460.38799999999998,"15min":7478.7280000000001},"last":19.134},{"interface":"front","average":{"1min":47.198,"5min":34.844999999999999,"15min":61.795000000000002},"min":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":7.125},"max":{"1min":470.99599999999998,"5min":470.99599999999998,"15min":7477.2579999999998},"last":17.117000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":44.006999999999998,"5min":37.201999999999998,"15min":29.158999999999999},"min":{"1min":13.180999999999999,"5min":12.316000000000001,"15min":8.7569999999999997},"max":{"1min":386.452,"5min":386.452,"15min":386.452},"last":17.783999999999999},{"interface":"front","average":{"1min":31.984999999999999,"5min":28.108000000000001,"15min":25.620999999999999},"min":{"1min":8.2970000000000006,"5min":7.835,"15min":7.835},"max":{"1min":300.58199999999999,"5min":300.58199999999999,"15min":300.58199999999999},"last":20.059000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:06 2024","interfaces":[{"interface":"back","average":{"1min":11.897,"5min":22.975999999999999,"15min":327.85500000000002},"min":{"1min":7.7320000000000002,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":18.007000000000001,"5min":295.38600000000002,"15min":21868.147000000001},"last":12.465},{"interface":"front","average":{"1min":13.711,"5min":24.344000000000001,"15min":331.459},"min":{"1min":10.34,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":18.896999999999998,"5min":302.11500000000001,"15min":21842.699000000001},"last":20.879000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346253,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78612,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293228,"statfs":{"total":96636764160,"available":96556265472,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":42.113,"5min":30.841000000000001,"15min":21.015000000000001},"min":{"1min":5.0019999999999998,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":193.56299999999999,"5min":350.36599999999999,"15min":350.36599999999999},"last":12.611000000000001},{"interface":"front","average":{"1min":39.423000000000002,"5min":29.962,"15min":26.541},"min":{"1min":6.9080000000000004,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":196.321,"5min":410.93099999999998,"15min":410.93099999999998},"last":10.202999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":26.478999999999999,"5min":24.071999999999999,"15min":20.161999999999999},"min":{"1min":5.5709999999999997,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":137.941,"5min":313.54399999999998,"15min":313.54399999999998},"last":8.5299999999999994},{"interface":"front","average":{"1min":33.582000000000001,"5min":33.255000000000003,"15min":26.905999999999999},"min":{"1min":8.0679999999999996,"5min":8.0679999999999996,"15min":6.3719999999999999},"max":{"1min":138.87200000000001,"5min":466.54599999999999,"15min":466.54599999999999},"last":17.248000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":36.649000000000001,"5min":24.920999999999999,"15min":25.821999999999999},"min":{"1min":7.6799999999999997,"5min":6.1139999999999999,"15min":6.1139999999999999},"max":{"1min":198.03100000000001,"5min":371.74200000000002,"15min":768.79100000000005},"last":9.3000000000000007},{"interface":"front","average":{"1min":41.881,"5min":29.448,"15min":25.213000000000001},"min":{"1min":9.2889999999999997,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":200.797,"5min":372.322,"15min":372.322},"last":13.851000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":3380.04,"5min":1086.2429999999999,"15min":410.63499999999999},"min":{"1min":9.2530000000000001,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":13684.860000000001,"5min":13684.860000000001,"15min":13684.860000000001},"last":16.497},{"interface":"front","average":{"1min":3346.288,"5min":1072.2090000000001,"15min":407.68000000000001},"min":{"1min":10.25,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":13604.335999999999,"5min":13604.335999999999,"15min":13604.335999999999},"last":11.429}]},{"osd":5,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":34.985999999999997,"5min":31.134,"15min":43.316000000000003},"min":{"1min":11.901999999999999,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":202.54400000000001,"5min":421.40600000000001,"15min":4807.1080000000002},"last":13.279999999999999},{"interface":"front","average":{"1min":32.462000000000003,"5min":31.606000000000002,"15min":46.908000000000001},"min":{"1min":9.0909999999999993,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":140.81399999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":7.71}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":35.414000000000001,"5min":31.486000000000001,"15min":30.466999999999999},"min":{"1min":9.8079999999999998,"5min":9.4640000000000004,"15min":9.4640000000000004},"max":{"1min":223.85499999999999,"5min":529.13699999999994,"15min":529.13699999999994},"last":14.413},{"interface":"front","average":{"1min":38.140999999999998,"5min":34.231999999999999,"15min":36.503999999999998},"min":{"1min":13.925000000000001,"5min":11.499000000000001,"15min":7.1870000000000003},"max":{"1min":223.16999999999999,"5min":529.73000000000002,"15min":967.58399999999995},"last":10.868}]},{"osd":7,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":41.027999999999999,"5min":36.396999999999998,"15min":341.108},"min":{"1min":11.250999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":219.53100000000001,"5min":530.32500000000005,"15min":20845.164000000001},"last":17.927},{"interface":"front","average":{"1min":39.600999999999999,"5min":46.209000000000003,"15min":345.48099999999999},"min":{"1min":9.8770000000000007,"5min":8.9100000000000001,"15min":7.5060000000000002},"max":{"1min":190.62799999999999,"5min":664.02300000000002,"15min":20888.814999999999},"last":16.045000000000002}]}]},{"osd":7,"up_from":19,"seq":81604378953,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60560,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311280,"statfs":{"total":96636764160,"available":96574750720,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":12.272,"5min":21.870999999999999,"15min":22.189},"min":{"1min":5.0609999999999999,"5min":4.8719999999999999,"15min":4.2789999999999999},"max":{"1min":51.43,"5min":358.94499999999999,"15min":368.16899999999998},"last":10.087},{"interface":"front","average":{"1min":11.212999999999999,"5min":25.902000000000001,"15min":24.701000000000001},"min":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":50.631,"5min":349.97199999999998,"15min":349.97199999999998},"last":9.3989999999999991}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":9.4149999999999991,"5min":14.821,"15min":24.390999999999998},"min":{"1min":4.577,"5min":4.577,"15min":4.577},"max":{"1min":16.036999999999999,"5min":94.453999999999994,"15min":472.33300000000003},"last":5.282},{"interface":"front","average":{"1min":11.449,"5min":21.518000000000001,"15min":26.239000000000001},"min":{"1min":7.4370000000000003,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":14.566000000000001,"5min":184.33500000000001,"15min":292.92099999999999},"last":13.220000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":15.978999999999999,"5min":27.795000000000002,"15min":31.538},"min":{"1min":8.4410000000000007,"5min":8.4410000000000007,"15min":6.6449999999999996},"max":{"1min":77.424000000000007,"5min":341.96100000000001,"15min":400.43099999999998},"last":16.710000000000001},{"interface":"front","average":{"1min":17.446000000000002,"5min":33.427,"15min":34.75},"min":{"1min":10.714,"5min":10.554,"15min":7.5049999999999999},"max":{"1min":53.476999999999997,"5min":389.37200000000001,"15min":439.31999999999999},"last":15.516999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":896.39800000000002,"5min":450.06999999999999,"15min":204.417},"min":{"1min":11.375999999999999,"5min":11.375999999999999,"15min":7.5940000000000003},"max":{"1min":13240.941000000001,"5min":13240.941000000001,"15min":13240.941000000001},"last":17.523},{"interface":"front","average":{"1min":915.66300000000001,"5min":450.815,"15min":204.88300000000001},"min":{"1min":7.484,"5min":7.423,"15min":7.423},"max":{"1min":13596.368,"5min":13596.368,"15min":13596.368},"last":13.984}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":11.497,"5min":349.65699999999998,"15min":225.608},"min":{"1min":7.0609999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":19.181999999999999,"5min":11792.466,"15min":13877.913},"last":19.887},{"interface":"front","average":{"1min":13.494999999999999,"5min":360.642,"15min":230.58600000000001},"min":{"1min":5.5119999999999996,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":19.454000000000001,"5min":11966.236999999999,"15min":14034.805},"last":12.462}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":24.239000000000001,"15min":108.086},"min":{"1min":11.356999999999999,"5min":7.6619999999999999,"15min":7.4050000000000002},"max":{"1min":21.242000000000001,"5min":345.90499999999997,"15min":7694.1369999999997},"last":19.100000000000001},{"interface":"front","average":{"1min":12.236000000000001,"5min":22.170000000000002,"15min":112.55800000000001},"min":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":18.193000000000001,"5min":203.31299999999999,"15min":7725.4070000000002},"last":10.975}]},{"osd":6,"last update":"Sat Mar 30 07:42:11 2024","interfaces":[{"interface":"back","average":{"1min":27.544,"5min":35.518999999999998,"15min":31.760999999999999},"min":{"1min":9.8190000000000008,"5min":8.7620000000000005,"15min":8.7620000000000005},"max":{"1min":171.96100000000001,"5min":316.012,"15min":316.012},"last":18.317},{"interface":"front","average":{"1min":35.613999999999997,"5min":36.930999999999997,"15min":29.056999999999999},"min":{"1min":18.420999999999999,"5min":6.8700000000000001,"15min":6.3090000000000002},"max":{"1min":194.72399999999999,"5min":743.27700000000004,"15min":743.27700000000004},"last":14.866}]}]},{"osd":1,"up_from":16,"seq":68719477079,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60952,"kb_used_data":7696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310888,"statfs":{"total":96636764160,"available":96574349312,"internally_reserved":0,"allocated":7880704,"data_stored":4551427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":13.853999999999999,"5min":12.869,"15min":18.533999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":55.762,"5min":132.02699999999999,"15min":421.78699999999998},"last":10.539},{"interface":"front","average":{"1min":12.318,"5min":12.644,"15min":18.782},"min":{"1min":7.5750000000000002,"5min":5.2759999999999998,"15min":5.2009999999999996},"max":{"1min":40.883000000000003,"5min":78.022999999999996,"15min":409.91000000000003},"last":5.7809999999999997}]},{"osd":2,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":9.1029999999999998,"5min":14.778,"15min":21.677},"min":{"1min":5.7489999999999997,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":18.859999999999999,"5min":180.42599999999999,"15min":433.34899999999999},"last":12.702999999999999},{"interface":"front","average":{"1min":11.426,"5min":18.071999999999999,"15min":23.861000000000001},"min":{"1min":8.8480000000000008,"5min":7.0700000000000003,"15min":5.6619999999999999},"max":{"1min":15.522,"5min":177.982,"15min":624.27300000000002},"last":15.885999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1671.011,"5min":557.96100000000001,"15min":225.822},"min":{"1min":9.8309999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":12913.004000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":15.074999999999999},{"interface":"front","average":{"1min":1712.2249999999999,"5min":566.452,"15min":227.315},"min":{"1min":9.1899999999999995,"5min":7.1689999999999996,"15min":6.6130000000000004},"max":{"1min":13136.255999999999,"5min":13136.255999999999,"15min":13136.255999999999},"last":16.805}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":11.67},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":13.951000000000001}]},{"osd":5,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":16.731000000000002,"5min":19.794,"15min":46.238},"min":{"1min":9.8879999999999999,"5min":9.8879999999999999,"15min":8.3439999999999994},"max":{"1min":32.831000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":17.766999999999999},{"interface":"front","average":{"1min":15.535,"5min":17.545999999999999,"15min":44.427999999999997},"min":{"1min":10.175000000000001,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":22.318999999999999,"5min":123.297,"15min":7345.2740000000003},"last":18.510000000000002}]},{"osd":6,"last update":"Sat Mar 30 07:41:46 2024","interfaces":[{"interface":"back","average":{"1min":17.646999999999998,"5min":19.326000000000001,"15min":26.009},"min":{"1min":11.462,"5min":9.2889999999999997,"15min":7.8540000000000001},"max":{"1min":24.370000000000001,"5min":137.40199999999999,"15min":318.84100000000001},"last":21.655999999999999},{"interface":"front","average":{"1min":12.010999999999999,"5min":13.401999999999999,"15min":24.765000000000001},"min":{"1min":8.1199999999999992,"5min":7.7110000000000003,"15min":7.4210000000000003},"max":{"1min":18.332000000000001,"5min":71.405000000000001,"15min":452.00700000000001},"last":7.6379999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:44:12 2024","interfaces":[{"interface":"back","average":{"1min":14.731999999999999,"5min":16.477,"15min":243.119},"min":{"1min":9.6280000000000001,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":20.175999999999998,"5min":115.461,"15min":20279.343000000001},"last":19.5},{"interface":"front","average":{"1min":12.869999999999999,"5min":22.413,"15min":251.726},"min":{"1min":6.4550000000000001,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":17.719999999999999,"5min":281.423,"15min":20311.815999999999},"last":12.023}]}]},{"osd":5,"up_from":231,"seq":992137445699,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":7.8410000000000002},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":11.460000000000001}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":14.939},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":8.4890000000000008}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":10.099},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":10.779999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":12.798},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":17.341999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":9.2609999999999992},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":13.696999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:42:38 2024","interfaces":[{"interface":"back","average":{"1min":30.094999999999999,"5min":22.585000000000001,"15min":27.227},"min":{"1min":14.298999999999999,"5min":11.044,"15min":8.9009999999999998},"max":{"1min":102.39400000000001,"5min":102.39400000000001,"15min":236.04599999999999},"last":16.675999999999998},{"interface":"front","average":{"1min":21.274000000000001,"5min":17.177,"15min":24.478999999999999},"min":{"1min":10.237,"5min":9.1620000000000008,"15min":6.4249999999999998},"max":{"1min":85.435000000000002,"5min":85.435000000000002,"15min":248.31299999999999},"last":18.577999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:44:13 2024","interfaces":[{"interface":"back","average":{"1min":11.587,"5min":15.007,"15min":283.82900000000001},"min":{"1min":7.585,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":18.324999999999999,"5min":115.589,"15min":19723.973000000002},"last":16.085999999999999},{"interface":"front","average":{"1min":15.435,"5min":19.620999999999999,"15min":283.815},"min":{"1min":10.228999999999999,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":19.495000000000001,"5min":95.783000000000001,"15min":19724.860000000001},"last":14.394}]}]},{"osd":3,"up_from":14,"seq":60129542479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82716,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289124,"statfs":{"total":96636764160,"available":96552062976,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":8.2729999999999997},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":6.9100000000000001}]},{"osd":1,"last update":"Sat Mar 30 07:44:28 2024","interfaces":[{"interface":"back","average":{"1min":34.786999999999999,"5min":22.965,"15min":21.632999999999999},"min":{"1min":8.6310000000000002,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":291.73599999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":11.064},{"interface":"front","average":{"1min":48.518999999999998,"5min":29.795999999999999,"15min":25.812000000000001},"min":{"1min":7.7089999999999996,"5min":7.5599999999999996,"15min":6.415},"max":{"1min":541.21199999999999,"5min":541.21199999999999,"15min":541.21199999999999},"last":11.372}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":14.712},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":13.569000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":11.663},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":12.882999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":15.558},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":17.067}]},{"osd":6,"last update":"Sat Mar 30 07:42:36 2024","interfaces":[{"interface":"back","average":{"1min":21.367999999999999,"5min":31.001999999999999,"15min":27.120000000000001},"min":{"1min":11.406000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":110.607,"5min":435.96600000000001,"15min":435.96600000000001},"last":16.132999999999999},{"interface":"front","average":{"1min":21.904,"5min":24.468,"15min":25.545999999999999},"min":{"1min":10.824,"5min":10.608000000000001,"15min":8.1180000000000003},"max":{"1min":62.828000000000003,"5min":145.239,"15min":259.60399999999998},"last":18.542999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":39.734000000000002,"5min":26.948,"15min":285.68700000000001},"min":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":8.2690000000000001},"max":{"1min":527.00300000000004,"5min":527.00300000000004,"15min":20495.457999999999},"last":17.978999999999999},{"interface":"front","average":{"1min":30.413,"5min":28.954000000000001,"15min":287.50799999999998},"min":{"1min":11.025,"5min":11.025,"15min":11.025},"max":{"1min":308.75900000000001,"5min":308.75900000000001,"15min":20496.241000000002},"last":19.704999999999998}]}]},{"osd":6,"up_from":978,"seq":4200478015813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60572,"kb_used_data":7304,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311268,"statfs":{"total":96636764160,"available":96574738432,"internally_reserved":0,"allocated":7479296,"data_stored":4153587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":463.66800000000001,"5min":104.768,"15min":47.588000000000001},"min":{"1min":463.66800000000001,"5min":5.8390000000000004,"15min":4.6150000000000002},"max":{"1min":463.66800000000001,"5min":463.66800000000001,"15min":463.66800000000001},"last":10.557},{"interface":"front","average":{"1min":466.404,"5min":108.209,"15min":52.970999999999997},"min":{"1min":466.404,"5min":8.5449999999999999,"15min":5.8150000000000004},"max":{"1min":466.404,"5min":466.404,"15min":466.404},"last":8.4789999999999992}]},{"osd":1,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":45.661999999999999,"5min":19.164999999999999,"15min":17.347000000000001},"min":{"1min":6.5149999999999997,"5min":5.4669999999999996,"15min":4.8579999999999997},"max":{"1min":298.72500000000002,"5min":298.72500000000002,"15min":298.72500000000002},"last":9.2449999999999992},{"interface":"front","average":{"1min":53.417000000000002,"5min":25.199999999999999,"15min":22.443000000000001},"min":{"1min":8.4190000000000005,"5min":7.944,"15min":7.5519999999999996},"max":{"1min":323.78800000000001,"5min":323.78800000000001,"15min":323.78800000000001},"last":13.452999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":314.89600000000002,"5min":76.939999999999998,"15min":39.972000000000001},"min":{"1min":314.89600000000002,"5min":6.351,"15min":6.351},"max":{"1min":314.89600000000002,"5min":314.89600000000002,"15min":314.89600000000002},"last":10.016},{"interface":"front","average":{"1min":353.24000000000001,"5min":87.257999999999996,"15min":46.240000000000002},"min":{"1min":353.24000000000001,"5min":8.266,"15min":8.266},"max":{"1min":353.24000000000001,"5min":353.24000000000001,"15min":353.24000000000001},"last":12.888999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:42:29 2024","interfaces":[{"interface":"back","average":{"1min":22.446999999999999,"5min":188.88,"15min":130.78899999999999},"min":{"1min":9.8559999999999999,"5min":6.7830000000000004,"15min":6.7830000000000004},"max":{"1min":144.38300000000001,"5min":9266.1270000000004,"15min":9266.1270000000004},"last":7.4740000000000002},{"interface":"front","average":{"1min":16.998000000000001,"5min":177.846,"15min":123.959},"min":{"1min":7.3179999999999996,"5min":6.1200000000000001,"15min":4.8339999999999996},"max":{"1min":86.635000000000005,"5min":9082.3240000000005,"15min":9082.3240000000005},"last":17.097000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":331.70499999999998,"5min":557.46400000000006,"15min":326.00999999999999},"min":{"1min":331.70499999999998,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":331.70499999999998,"5min":13752.589,"15min":13752.589},"last":11.141},{"interface":"front","average":{"1min":469.358,"5min":575.79200000000003,"15min":334.21499999999997},"min":{"1min":469.358,"5min":7.5999999999999996,"15min":7.0199999999999996},"max":{"1min":469.358,"5min":13613.244000000001,"15min":13613.244000000001},"last":6.1109999999999998}]},{"osd":5,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":42.067999999999998,"5min":19.300999999999998,"15min":110.551},"min":{"1min":8.1010000000000009,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":483.904,"5min":483.904,"15min":9617.3179999999993},"last":11.725},{"interface":"front","average":{"1min":46.238,"5min":24.486000000000001,"15min":110.789},"min":{"1min":9.4190000000000005,"5min":9.4190000000000005,"15min":8.8580000000000005},"max":{"1min":485.661,"5min":485.661,"15min":9386.0400000000009},"last":14.249000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:14 2024","interfaces":[{"interface":"back","average":{"1min":67.293000000000006,"5min":34.445,"15min":29.379999999999999},"min":{"1min":12.813000000000001,"5min":7.9729999999999999,"15min":7.9729999999999999},"max":{"1min":470.34300000000002,"5min":470.34300000000002,"15min":470.34300000000002},"last":12.35},{"interface":"front","average":{"1min":70.335999999999999,"5min":35.420000000000002,"15min":29.364000000000001},"min":{"1min":12.48,"5min":8.452,"15min":7.4189999999999996},"max":{"1min":487.30700000000002,"5min":487.30700000000002,"15min":487.30700000000002},"last":14.826000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.9 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.920 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.921 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 16.10 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-30T07:16:48.116623+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=16, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.922 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.923 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.924 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.924 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.924 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:00.924 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-30T07:45:02.202 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.013 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.204 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.015 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.223 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.033 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.258 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.068 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.258 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.035 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.259 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.036 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.260 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.071 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.273 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.050 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.276 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.086 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.962 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.739 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.963 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.740 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.971 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:30.748 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.980 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.791 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.982 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.792 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:02.995 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:30.805 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:03.012 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.823 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:03.013 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.824 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:03.022 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:30.832 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.300 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:45.111 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.302 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:45.113 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.319 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:45.130 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.355 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:45.132 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.356 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:45.133 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.367 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:45.178 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.370 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:45.180 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.374 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:45.151 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:17.387 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:45.198 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:20.924 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.023+0000 7f4f8e78e640 10 monclient: get_monmap_and_config 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.023+0000 7f4f8e78e640 10 monclient: build_initial_monmap 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.023+0000 7f4f8e78e640 10 monclient: monmap: 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:45:21.025106+0000 2024-03-30T07:45:21.026 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:45:21.025106+0000 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:45:21.027 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 Processor -- start 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 -- start start 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient: init 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f88073680 addr v1:172.21.15.47:6789/0 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f4f880af060 0x7f4f88144b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-c con 0x7f4f880af060 addr v2:172.21.15.52:3300/0 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f8813ef60 addr v1:172.21.15.52:6789/0 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8806e680 con 0x7f4f88073680 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f880afc20 con 0x7f4f8813ef60 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f88145780 con 0x7f4f880af060 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:21.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.024+0000 7f4f8e78e640 10 monclient(hunting): authenticate will time out at 3247.140896s 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.025+0000 7f4f87fff640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4f88073680 0x7f4f8813e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52326/0 (socket says 172.21.15.47:52326) 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.025+0000 7f4f87fff640 1 -- 172.21.15.47:0/933817677 learned_addr learned my addr 172.21.15.47:0/933817677 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.025+0000 7f4f877fe640 1 --2- 172.21.15.47:0/933817677 >> v2:172.21.15.52:3300/0 conn(0x7f4f880af060 0x7f4f88144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.026+0000 7f4f877fe640 10 monclient(hunting): get_auth_request con 0x7f4f880af060 auth_method 0 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.026+0000 7f4f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.026+0000 7f4f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.026+0000 7f4f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412434448 0 0) 0x7f4f8806e680 con 0x7f4f88073680 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 10 monclient(hunting): _negotiate 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 10 monclient(hunting): my global_id is 15930 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f70003520 con 0x7f4f88073680 2024-03-30T07:45:21.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:21.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:21.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.028+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223361322 0 0) 0x7f4f70003520 con 0x7f4f88073680 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 >> v2:172.21.15.52:3300/0 conn(0x7f4f880af060 msgr2=0x7f4f88144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 --2- 172.21.15.47:0/933817677 >> v2:172.21.15.52:3300/0 conn(0x7f4f880af060 0x7f4f88144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 >> v1:172.21.15.52:6789/0 conn(0x7f4f8813ef60 legacy=0x7f4f88145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _finish_hunting 0 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 monclient: found mon.noname-a-legacy 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.47:6789/0 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f880af760 con 0x7f4f88073680 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _finish_auth 0 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _check_auth_tickets 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:21.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 700119255 0 0) 0x7f4f78001670 con 0x7f4f88073680 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f8e78e640 5 monclient: authenticate success, global_id 15930 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: dump: 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:21.036 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] but i'm connected to v1:172.21.15.47:6789/0, reconnecting 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 >> v1:172.21.15.47:6789/0 conn(0x7f4f88073680 legacy=0x7f4f8813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700039c0 0x7f4f70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): picked mon.b con 0x7f4f700039c0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f700047c0 0x7f4f70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): picked mon.a con 0x7f4f700047c0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 0x7f4f7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): picked mon.c con 0x7f4f700051d0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7000c0f0 con 0x7f4f700047c0 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7000c260 con 0x7f4f700039c0 2024-03-30T07:45:21.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7000c420 con 0x7f4f700051d0 2024-03-30T07:45:21.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:21.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-30T07:45:21.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.031+0000 7f4f8e78e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-30T07:45:21.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f8cd2b640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 0x7f4f7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f8cd2b640 10 monclient(hunting): get_auth_request con 0x7f4f700051d0 auth_method 0 2024-03-30T07:45:21.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f8cd2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f8cd2b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f8cd2b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.032+0000 7f4f877fe640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f700047c0 0x7f4f70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.033+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.033+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.033+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f87fff640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700039c0 0x7f4f70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f877fe640 10 monclient(hunting): get_auth_request con 0x7f4f700047c0 auth_method 0 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_done global_id 15930 payload 293 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.034+0000 7f4f8cd2b640 1 -- 172.21.15.47:0/933817677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700039c0 msgr2=0x7f4f70003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700039c0 0x7f4f70003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 -- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f700047c0 msgr2=0x7f4f70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:21.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f700047c0 0x7f4f70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 10 monclient: _finish_hunting 0 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 monclient: found mon.c 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 -- 172.21.15.47:0/933817677 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f70003520 con 0x7f4f700051d0 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 10 monclient: _finish_auth 0 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 10 monclient: _check_auth_tickets 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.035+0000 7f4f8cd2b640 1 -- 172.21.15.47:0/933817677 reap_dead start 2024-03-30T07:45:21.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.036+0000 7f4f8cd2b640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 0x7f4f7000b980 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f4f880ade30 tx=0x7f4f700057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d069c24e5d1c9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:21.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f80003b70 con 0x7f4f700051d0 2024-03-30T07:45:21.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:21.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:21.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: dump: 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f80003cd0 con 0x7f4f700051d0 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 1 -- 172.21.15.47:0/933817677 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f800052f0 con 0x7f4f700051d0 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f8e78e640 10 monclient: get_monmap_and_config success 2024-03-30T07:45:21.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f8e78e640 4 set_mon_vals no callback set 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.040+0000 7f4f86ffd640 10 monclient: dump: 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.045 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:21.046 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:21.046 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:21.046 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 10 monclient: shutdown 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 -- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 msgr2=0x7f4f7000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 0x7f4f7000b980 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7f4f880ade30 tx=0x7f4f700057f0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 -- 172.21.15.47:0/933817677 shutdown_connections 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/933817677 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f700051d0 0x7f4f7000b980 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 -- 172.21.15.47:0/933817677 >> 172.21.15.47:0/933817677 conn(0x7f4f88069ec0 msgr2=0x7f4f8806a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 -- 172.21.15.47:0/933817677 shutdown_connections 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.041+0000 7f4f8e78e640 1 -- 172.21.15.47:0/933817677 wait complete. 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 10 monclient: build_initial_monmap 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 10 monclient: monmap: 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:21.049 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 librados: starting msgr at 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 librados: starting objecter 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 Processor -- start 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 -- start start 2024-03-30T07:45:21.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 librados: setting wanted keys 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 1 librados: calling monclient init 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.042+0000 7f4f8e78e640 10 monclient: init 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700051d0 0x7f4f88070000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-b con 0x7f4f700051d0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-c con 0x7f4f88140d00 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f881d99e0 0x7f4f881d9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): picked mon.noname-a con 0x7f4f881d99e0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88141f10 con 0x7f4f881d99e0 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f88141a80 con 0x7f4f700051d0 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f881416d0 con 0x7f4f88140d00 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8e78e640 10 monclient(hunting): authenticate will time out at 3247.159385s 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.043+0000 7f4f8cd2b640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f881d99e0 0x7f4f881d9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700051d0 0x7f4f88070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:53322/0 (socket says 172.21.15.47:53322) 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 1 -- 172.21.15.47:0/4081409862 learned_addr learned my addr 172.21.15.47:0/4081409862 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:45:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): get_auth_request con 0x7f4f88140d00 auth_method 0 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): get_auth_request con 0x7f4f881d99e0 auth_method 0 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): get_auth_request con 0x7f4f700051d0 auth_method 0 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f8cd2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:21.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.044+0000 7f4f87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:21.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 10 monclient(hunting): handle_auth_done global_id 17144 payload 824 2024-03-30T07:45:21.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 -- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700051d0 msgr2=0x7f4f88070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700051d0 0x7f4f88070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 -- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f881d99e0 msgr2=0x7f4f881d9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f881d99e0 0x7f4f881d9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 10 monclient: _finish_hunting 0 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 monclient: found mon.noname-c 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.047+0000 7f4f877fe640 1 -- 172.21.15.47:0/4081409862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f881da3d0 con 0x7f4f88140d00 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.049+0000 7f4f877fe640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0115a0 tx=0x7f4f7c011a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.049+0000 7f4f84ff9640 10 client.?.objecter ms_handle_connect 0x7f4f88140d00 2024-03-30T07:45:21.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.049+0000 7f4f84ff9640 10 client.?.objecter resend_mon_ops 2024-03-30T07:45:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c015930 con 0x7f4f88140d00 2024-03-30T07:45:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:45:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: dump: 2024-03-30T07:45:21.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: _finish_auth 0 2024-03-30T07:45:21.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: _check_auth_tickets 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c015a90 con 0x7f4f88140d00 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 5 monclient: authenticate success, global_id 17144 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 10 monclient: _renew_subs 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f881d64a0 con 0x7f4f88140d00 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c015e20 con 0x7f4f88140d00 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f84ff9640 10 monclient: dump: 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:21.060 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f867fc640 4 set_mon_vals no callback set 2024-03-30T07:45:21.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 10 client.17144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 10 monclient: _renew_subs 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f881d69b0 con 0x7f4f88140d00 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.053+0000 7f4f8e78e640 1 librados: init done 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.054+0000 7f4f8e78e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.054+0000 7f4f8e78e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.054+0000 7f4f8e78e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:21.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.054+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f881da620 con 0x7f4f88140d00 2024-03-30T07:45:21.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.073+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f7c005990 con 0x7f4f88140d00 2024-03-30T07:45:21.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.073+0000 7f4f84ff9640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 0x7f4f7406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:21.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.074+0000 7f4f87fff640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 0x7f4f7406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:21.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.074+0000 7f4f87fff640 10 monclient: get_auth_request con 0x7f4f7406c5e0 auth_method 0 2024-03-30T07:45:21.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.074+0000 7f4f84ff9640 10 client.17144.objecter ms_handle_connect 0x7f4f7406c5e0 2024-03-30T07:45:21.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.074+0000 7f4f87fff640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 0x7f4f7406eaa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f78000c00 tx=0x7f4f78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.329+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f4f7c059100 con 0x7f4f88140d00 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.329+0000 7f4f84ff9640 10 client.17144.objecter ms_dispatch 0x7f4f88000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.329+0000 7f4f84ff9640 3 client.17144.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.329+0000 7f4f84ff9640 3 client.17144.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.330+0000 7f4f84ff9640 20 client.17144.objecter dump_active .. 0 homeless 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.330+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f4f7c055440 con 0x7f4f88140d00 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.330+0000 7f4f84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.330+0000 7f4f84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:45:21.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.512+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f88073a60 con 0x7f4f7406c5e0 2024-03-30T07:45:21.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.517+0000 7f4f84ff9640 1 -- 172.21.15.47:0/4081409862 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152521 (secure 0 0 0) 0x7f4f7800d050 con 0x7f4f7406c5e0 2024-03-30T07:45:21.522 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:45:21.522 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:45:21.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 10 librados: watch_flush enter 2024-03-30T07:45:21.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 10 librados: watch_flush exit 2024-03-30T07:45:21.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 20 client.17144.objecter shutdown clearing up homeless session... 2024-03-30T07:45:21.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 10 client.17144.objecter successfully canceled tick 2024-03-30T07:45:21.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 msgr2=0x7f4f7406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 0x7f4f7406eaa0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f78000c00 tx=0x7f4f78009040 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 10 monclient: shutdown 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 msgr2=0x7f4f88141190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0115a0 tx=0x7f4f7c011a60 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 shutdown_connections 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f4f7406c5e0 0x7f4f7406eaa0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f881d99e0 0x7f4f881d9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f88140d00 0x7f4f88141190 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 --2- 172.21.15.47:0/4081409862 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f4f700051d0 0x7f4f88070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 >> 172.21.15.47:0/4081409862 conn(0x7f4f88069ec0 msgr2=0x7f4f8813cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 shutdown_connections 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 -- 172.21.15.47:0/4081409862 wait complete. 2024-03-30T07:45:21.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:21.520+0000 7f4f8e78e640 1 librados: shutdown 2024-03-30T07:45:21.534 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1863,"stamp":"2024-03-30T07:45:20.377941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529528,"kb_used_data":59192,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445192,"statfs":{"total":773094113280,"available":772551876608,"internally_reserved":0,"allocated":60612608,"data_stored":34024346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002608"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.100288+0000","last_change":"2024-03-30T07:45:03.102050+0000","last_active":"2024-03-30T07:45:03.100288+0000","last_peered":"2024-03-30T07:45:03.100288+0000","last_clean":"2024-03-30T07:45:03.100288+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:45:03.100288+0000","last_undegraded":"2024-03-30T07:45:03.100288+0000","last_fullsized":"2024-03-30T07:45:03.100288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.099490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.099490+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.099490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:52.186954+0000","scrub_duration":0.044015481000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":958,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.136050+0000","last_change":"2024-03-30T07:45:04.137799+0000","last_active":"2024-03-30T07:45:04.136050+0000","last_peered":"2024-03-30T07:45:04.136050+0000","last_clean":"2024-03-30T07:45:04.136050+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:45:04.136050+0000","last_undegraded":"2024-03-30T07:45:04.136050+0000","last_fullsized":"2024-03-30T07:45:04.136050+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.135262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.135262+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.135262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:11:25.177760+0000","scrub_duration":0.075101082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":933,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:11.563546+0000","last_change":"2024-03-30T07:45:11.565294+0000","last_active":"2024-03-30T07:45:11.563546+0000","last_peered":"2024-03-30T07:45:11.563546+0000","last_clean":"2024-03-30T07:45:11.563546+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:45:11.563546+0000","last_undegraded":"2024-03-30T07:45:11.563546+0000","last_fullsized":"2024-03-30T07:45:11.563546+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:11.562727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:11.562727+0000","last_clean_scrub_stamp":"2024-03-30T07:45:11.562727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:13:02.794032+0000","scrub_duration":0.033870826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:05.286114+0000","last_change":"2024-03-30T07:45:05.286114+0000","last_active":"2024-03-30T07:45:05.286114+0000","last_peered":"2024-03-30T07:45:05.286114+0000","last_clean":"2024-03-30T07:45:05.286114+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:45:05.286114+0000","last_undegraded":"2024-03-30T07:45:05.286114+0000","last_fullsized":"2024-03-30T07:45:05.286114+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:05.279442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:05.279442+0000","last_clean_scrub_stamp":"2024-03-30T07:45:05.279442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:51:04.595437+0000","scrub_duration":0.073014201000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":925,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.857888+0000","last_change":"2024-03-30T07:45:03.859580+0000","last_active":"2024-03-30T07:45:03.857888+0000","last_peered":"2024-03-30T07:45:03.857888+0000","last_clean":"2024-03-30T07:45:03.857888+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:45:03.857888+0000","last_undegraded":"2024-03-30T07:45:03.857888+0000","last_fullsized":"2024-03-30T07:45:03.857888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.857105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.857105+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.857105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:45.010470+0000","scrub_duration":0.034940751999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":936,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.891505+0000","last_change":"2024-03-30T07:45:04.893251+0000","last_active":"2024-03-30T07:45:04.891505+0000","last_peered":"2024-03-30T07:45:04.891505+0000","last_clean":"2024-03-30T07:45:04.891505+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:45:04.891505+0000","last_undegraded":"2024-03-30T07:45:04.891505+0000","last_fullsized":"2024-03-30T07:45:04.891505+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.890711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.890711+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.890711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:43:45.934457+0000","scrub_duration":0.051333487999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":949,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.574165+0000","last_change":"2024-03-30T07:45:03.575953+0000","last_active":"2024-03-30T07:45:03.574165+0000","last_peered":"2024-03-30T07:45:03.574165+0000","last_clean":"2024-03-30T07:45:03.574165+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:45:03.574165+0000","last_undegraded":"2024-03-30T07:45:03.574165+0000","last_fullsized":"2024-03-30T07:45:03.574165+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.573356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.573356+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.573356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:59:20.825982+0000","scrub_duration":0.047055578000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":935,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:07.546857+0000","last_change":"2024-03-30T07:45:07.548601+0000","last_active":"2024-03-30T07:45:07.546857+0000","last_peered":"2024-03-30T07:45:07.546857+0000","last_clean":"2024-03-30T07:45:07.546857+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:45:07.546857+0000","last_undegraded":"2024-03-30T07:45:07.546857+0000","last_fullsized":"2024-03-30T07:45:07.546857+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:07.546069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:07.546069+0000","last_clean_scrub_stamp":"2024-03-30T07:45:07.546069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:17:25.702685+0000","scrub_duration":0.073536952000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":938,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.961851+0000","last_change":"2024-03-30T07:45:00.961851+0000","last_active":"2024-03-30T07:45:00.961851+0000","last_peered":"2024-03-30T07:45:00.961851+0000","last_clean":"2024-03-30T07:45:00.961851+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:45:00.961851+0000","last_undegraded":"2024-03-30T07:45:00.961851+0000","last_fullsized":"2024-03-30T07:45:00.961851+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:00.959383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:00.959383+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.959383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:36:05.184291+0000","scrub_duration":0.042599705000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:59.934894+0000","last_change":"2024-03-30T07:44:59.936597+0000","last_active":"2024-03-30T07:44:59.934894+0000","last_peered":"2024-03-30T07:44:59.934894+0000","last_clean":"2024-03-30T07:44:59.934894+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:59.934894+0000","last_undegraded":"2024-03-30T07:44:59.934894+0000","last_fullsized":"2024-03-30T07:44:59.934894+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:59.934115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:59.934115+0000","last_clean_scrub_stamp":"2024-03-30T07:44:59.934115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:42:05.163853+0000","scrub_duration":0.052884168000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":927,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.677765+0000","last_change":"2024-03-30T07:24:13.737991+0000","last_active":"2024-03-30T07:44:49.677765+0000","last_peered":"2024-03-30T07:44:49.677765+0000","last_clean":"2024-03-30T07:44:49.677765+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:44:49.677765+0000","last_undegraded":"2024-03-30T07:44:49.677765+0000","last_fullsized":"2024-03-30T07:44:49.677765+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.948801+0000","last_change":"2024-03-30T07:45:01.959948+0000","last_active":"2024-03-30T07:45:01.948801+0000","last_peered":"2024-03-30T07:45:01.948801+0000","last_clean":"2024-03-30T07:45:01.948801+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:45:01.948801+0000","last_undegraded":"2024-03-30T07:45:01.948801+0000","last_fullsized":"2024-03-30T07:45:01.948801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.948015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.948015+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.948015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:07:50.406960+0000","scrub_duration":0.075824290000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.762320+0000","last_change":"2024-03-30T07:45:01.769055+0000","last_active":"2024-03-30T07:45:01.762320+0000","last_peered":"2024-03-30T07:45:01.762320+0000","last_clean":"2024-03-30T07:45:01.762320+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:45:01.762320+0000","last_undegraded":"2024-03-30T07:45:01.762320+0000","last_fullsized":"2024-03-30T07:45:01.762320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.761515+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:32.025372+0000","scrub_duration":0.054478171999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.621808+0000","last_change":"2024-03-30T07:45:02.624117+0000","last_active":"2024-03-30T07:45:02.621808+0000","last_peered":"2024-03-30T07:45:02.621808+0000","last_clean":"2024-03-30T07:45:02.621808+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:45:02.621808+0000","last_undegraded":"2024-03-30T07:45:02.621808+0000","last_fullsized":"2024-03-30T07:45:02.621808+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.620958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.620958+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.620958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:47:25.436827+0000","scrub_duration":0.038212127999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:57.908790+0000","last_change":"2024-03-30T07:44:57.910639+0000","last_active":"2024-03-30T07:44:57.908790+0000","last_peered":"2024-03-30T07:44:57.908790+0000","last_clean":"2024-03-30T07:44:57.908790+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:57.908790+0000","last_undegraded":"2024-03-30T07:44:57.908790+0000","last_fullsized":"2024-03-30T07:44:57.908790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:57.907974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:57.907974+0000","last_clean_scrub_stamp":"2024-03-30T07:44:57.907974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:10:30.858711+0000","scrub_duration":0.04023471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:05.848774+0000","last_change":"2024-03-30T07:45:05.850948+0000","last_active":"2024-03-30T07:45:05.848774+0000","last_peered":"2024-03-30T07:45:05.848774+0000","last_clean":"2024-03-30T07:45:05.848774+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:45:05.848774+0000","last_undegraded":"2024-03-30T07:45:05.848774+0000","last_fullsized":"2024-03-30T07:45:05.848774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:05.847955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:05.847955+0000","last_clean_scrub_stamp":"2024-03-30T07:45:05.847955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:22:18.393244+0000","scrub_duration":0.031780224000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:08.737456+0000","last_change":"2024-03-30T07:45:08.743279+0000","last_active":"2024-03-30T07:45:08.737456+0000","last_peered":"2024-03-30T07:45:08.737456+0000","last_clean":"2024-03-30T07:45:08.737456+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:45:08.737456+0000","last_undegraded":"2024-03-30T07:45:08.737456+0000","last_fullsized":"2024-03-30T07:45:08.737456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:08.736649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:08.736649+0000","last_clean_scrub_stamp":"2024-03-30T07:45:08.736649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:28:46.083314+0000","scrub_duration":0.037436690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":922,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:11.966673+0000","last_change":"2024-03-30T07:45:11.968405+0000","last_active":"2024-03-30T07:45:11.966673+0000","last_peered":"2024-03-30T07:45:11.966673+0000","last_clean":"2024-03-30T07:45:11.966673+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:45:11.966673+0000","last_undegraded":"2024-03-30T07:45:11.966673+0000","last_fullsized":"2024-03-30T07:45:11.966673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:11.965874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:11.965874+0000","last_clean_scrub_stamp":"2024-03-30T07:45:11.965874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:06:09.430315+0000","scrub_duration":0.080360671999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":965,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.226721+0000","last_change":"2024-03-30T07:45:02.228821+0000","last_active":"2024-03-30T07:45:02.226721+0000","last_peered":"2024-03-30T07:45:02.226721+0000","last_clean":"2024-03-30T07:45:02.226721+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:45:02.226721+0000","last_undegraded":"2024-03-30T07:45:02.226721+0000","last_fullsized":"2024-03-30T07:45:02.226721+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.225916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.225916+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.225916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:12:30.187116+0000","scrub_duration":0.055995208999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.578682+0000","last_change":"2024-03-30T07:45:03.580413+0000","last_active":"2024-03-30T07:45:03.578682+0000","last_peered":"2024-03-30T07:45:03.578682+0000","last_clean":"2024-03-30T07:45:03.578682+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:45:03.578682+0000","last_undegraded":"2024-03-30T07:45:03.578682+0000","last_fullsized":"2024-03-30T07:45:03.578682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.577615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.577615+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.577615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:38:39.511514+0000","scrub_duration":0.034900265999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":946,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:15.644763+0000","last_change":"2024-03-30T07:45:15.646390+0000","last_active":"2024-03-30T07:45:15.644763+0000","last_peered":"2024-03-30T07:45:15.644763+0000","last_clean":"2024-03-30T07:45:15.644763+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:45:15.644763+0000","last_undegraded":"2024-03-30T07:45:15.644763+0000","last_fullsized":"2024-03-30T07:45:15.644763+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:15.643975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:15.643975+0000","last_clean_scrub_stamp":"2024-03-30T07:45:15.643975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:38:18.694937+0000","scrub_duration":0.031838125000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":940,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.469210+0000","last_change":"2024-03-30T07:45:01.470886+0000","last_active":"2024-03-30T07:45:01.469210+0000","last_peered":"2024-03-30T07:45:01.469210+0000","last_clean":"2024-03-30T07:45:01.469210+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:45:01.469210+0000","last_undegraded":"2024-03-30T07:45:01.469210+0000","last_fullsized":"2024-03-30T07:45:01.469210+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.468400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.468400+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.468400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:34:12.252367+0000","scrub_duration":0.029422416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":956,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.819965+0000","last_change":"2024-03-30T07:45:00.828787+0000","last_active":"2024-03-30T07:45:00.819965+0000","last_peered":"2024-03-30T07:45:00.819965+0000","last_clean":"2024-03-30T07:45:00.819965+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:45:00.819965+0000","last_undegraded":"2024-03-30T07:45:00.819965+0000","last_fullsized":"2024-03-30T07:45:00.819965+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:00.817708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:00.817708+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.817708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:21:03.691881+0000","scrub_duration":0.20702821599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":934,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:06.531324+0000","last_change":"2024-03-30T07:45:06.533065+0000","last_active":"2024-03-30T07:45:06.531324+0000","last_peered":"2024-03-30T07:45:06.531324+0000","last_clean":"2024-03-30T07:45:06.531324+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:45:06.531324+0000","last_undegraded":"2024-03-30T07:45:06.531324+0000","last_fullsized":"2024-03-30T07:45:06.531324+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:06.530509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:06.530509+0000","last_clean_scrub_stamp":"2024-03-30T07:45:06.530509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:44:12.438188+0000","scrub_duration":0.048993979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:57.719569+0000","last_change":"2024-03-30T07:44:57.719569+0000","last_active":"2024-03-30T07:44:57.719569+0000","last_peered":"2024-03-30T07:44:57.719569+0000","last_clean":"2024-03-30T07:44:57.719569+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:57.719569+0000","last_undegraded":"2024-03-30T07:44:57.719569+0000","last_fullsized":"2024-03-30T07:44:57.719569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:57.716513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:57.716513+0000","last_clean_scrub_stamp":"2024-03-30T07:44:57.716513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:14:29.808330+0000","scrub_duration":0.042252514999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":957,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:59.102794+0000","last_change":"2024-03-30T07:44:59.108606+0000","last_active":"2024-03-30T07:44:59.102794+0000","last_peered":"2024-03-30T07:44:59.102794+0000","last_clean":"2024-03-30T07:44:59.102794+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:59.102794+0000","last_undegraded":"2024-03-30T07:44:59.102794+0000","last_fullsized":"2024-03-30T07:44:59.102794+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:59.101982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:59.101982+0000","last_clean_scrub_stamp":"2024-03-30T07:44:59.101982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:03.681571+0000","scrub_duration":0.034080829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:58.742553+0000","last_change":"2024-03-30T07:44:58.744347+0000","last_active":"2024-03-30T07:44:58.742553+0000","last_peered":"2024-03-30T07:44:58.742553+0000","last_clean":"2024-03-30T07:44:58.742553+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:58.742553+0000","last_undegraded":"2024-03-30T07:44:58.742553+0000","last_fullsized":"2024-03-30T07:44:58.742553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:58.741755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:58.741755+0000","last_clean_scrub_stamp":"2024-03-30T07:44:58.741755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:27:46.255761+0000","scrub_duration":0.051197923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"994'78","reported_seq":1091,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:56.280200+0000","last_change":"2024-03-30T07:44:56.282541+0000","last_active":"2024-03-30T07:44:56.280200+0000","last_peered":"2024-03-30T07:44:56.280200+0000","last_clean":"2024-03-30T07:44:56.280200+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:56.280200+0000","last_undegraded":"2024-03-30T07:44:56.280200+0000","last_fullsized":"2024-03-30T07:44:56.280200+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"994'78","last_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_deep_scrub":"994'78","last_deep_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_clean_scrub_stamp":"2024-03-30T07:44:56.279300+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:20:04.428365+0000","scrub_duration":0.11938407600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":962,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.973792+0000","last_change":"2024-03-30T07:45:01.980356+0000","last_active":"2024-03-30T07:45:01.973792+0000","last_peered":"2024-03-30T07:45:01.973792+0000","last_clean":"2024-03-30T07:45:01.973792+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:45:01.973792+0000","last_undegraded":"2024-03-30T07:45:01.973792+0000","last_fullsized":"2024-03-30T07:45:01.973792+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.971705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.971705+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.971705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:57:44.662538+0000","scrub_duration":0.050853467999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":969,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.172924+0000","last_change":"2024-03-30T07:45:02.174685+0000","last_active":"2024-03-30T07:45:02.172924+0000","last_peered":"2024-03-30T07:45:02.172924+0000","last_clean":"2024-03-30T07:45:02.172924+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:45:02.172924+0000","last_undegraded":"2024-03-30T07:45:02.172924+0000","last_fullsized":"2024-03-30T07:45:02.172924+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.172128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.172128+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.172128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:32:49.128070+0000","scrub_duration":0.079233095000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":939,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.559831+0000","last_change":"2024-03-30T07:45:04.561495+0000","last_active":"2024-03-30T07:45:04.559831+0000","last_peered":"2024-03-30T07:45:04.559831+0000","last_clean":"2024-03-30T07:45:04.559831+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:45:04.559831+0000","last_undegraded":"2024-03-30T07:45:04.559831+0000","last_fullsized":"2024-03-30T07:45:04.559831+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.559009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.559009+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.559009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:16:59.538322+0000","scrub_duration":0.065331189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":914,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:52.454172+0000","last_change":"2024-03-30T07:18:04.168467+0000","last_active":"2024-03-30T07:44:52.454172+0000","last_peered":"2024-03-30T07:44:52.454172+0000","last_clean":"2024-03-30T07:44:52.454172+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:44:52.454172+0000","last_undegraded":"2024-03-30T07:44:52.454172+0000","last_fullsized":"2024-03-30T07:44:52.454172+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":948,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.661133+0000","last_change":"2024-03-30T07:45:01.667042+0000","last_active":"2024-03-30T07:45:01.661133+0000","last_peered":"2024-03-30T07:45:01.661133+0000","last_clean":"2024-03-30T07:45:01.661133+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:45:01.661133+0000","last_undegraded":"2024-03-30T07:45:01.661133+0000","last_fullsized":"2024-03-30T07:45:01.661133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.660355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.660355+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.660355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:49:18.855837+0000","scrub_duration":0.041866140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:06.144826+0000","last_change":"2024-03-30T07:45:06.150467+0000","last_active":"2024-03-30T07:45:06.144826+0000","last_peered":"2024-03-30T07:45:06.144826+0000","last_clean":"2024-03-30T07:45:06.144826+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:45:06.144826+0000","last_undegraded":"2024-03-30T07:45:06.144826+0000","last_fullsized":"2024-03-30T07:45:06.144826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:06.144033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:06.144033+0000","last_clean_scrub_stamp":"2024-03-30T07:45:06.144033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:39.885228+0000","scrub_duration":0.083749650999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":964,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.953189+0000","last_change":"2024-03-30T07:45:02.955292+0000","last_active":"2024-03-30T07:45:02.953189+0000","last_peered":"2024-03-30T07:45:02.953189+0000","last_clean":"2024-03-30T07:45:02.953189+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:45:02.953189+0000","last_undegraded":"2024-03-30T07:45:02.953189+0000","last_fullsized":"2024-03-30T07:45:02.953189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.952382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.952382+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.952382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:26:26.189127+0000","scrub_duration":0.037306020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":921,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:10.906081+0000","last_change":"2024-03-30T07:45:10.907928+0000","last_active":"2024-03-30T07:45:10.906081+0000","last_peered":"2024-03-30T07:45:10.906081+0000","last_clean":"2024-03-30T07:45:10.906081+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:45:10.906081+0000","last_undegraded":"2024-03-30T07:45:10.906081+0000","last_fullsized":"2024-03-30T07:45:10.906081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:10.905264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:10.905264+0000","last_clean_scrub_stamp":"2024-03-30T07:45:10.905264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:58:09.541385+0000","scrub_duration":0.054738342000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":922,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:08.853422+0000","last_change":"2024-03-30T07:45:08.856035+0000","last_active":"2024-03-30T07:45:08.853422+0000","last_peered":"2024-03-30T07:45:08.853422+0000","last_clean":"2024-03-30T07:45:08.853422+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:45:08.853422+0000","last_undegraded":"2024-03-30T07:45:08.853422+0000","last_fullsized":"2024-03-30T07:45:08.853422+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:08.852576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:08.852576+0000","last_clean_scrub_stamp":"2024-03-30T07:45:08.852576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:57:33.949616+0000","scrub_duration":0.032762352000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":915,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:49.669547+0000","last_change":"2024-03-30T07:18:03.197722+0000","last_active":"2024-03-30T07:44:49.669547+0000","last_peered":"2024-03-30T07:44:49.669547+0000","last_clean":"2024-03-30T07:44:49.669547+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:44:49.669547+0000","last_undegraded":"2024-03-30T07:44:49.669547+0000","last_fullsized":"2024-03-30T07:44:49.669547+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:17:52.426377+0000","last_clean_scrub_stamp":"2024-03-30T07:17:52.426377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":959,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.079416+0000","last_change":"2024-03-30T07:45:02.085105+0000","last_active":"2024-03-30T07:45:02.079416+0000","last_peered":"2024-03-30T07:45:02.079416+0000","last_clean":"2024-03-30T07:45:02.079416+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:45:02.079416+0000","last_undegraded":"2024-03-30T07:45:02.079416+0000","last_fullsized":"2024-03-30T07:45:02.079416+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.078616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.078616+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.078616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:10.960802+0000","scrub_duration":0.063356277000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"994'8","reported_seq":955,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.624731+0000","last_change":"2024-03-30T07:45:00.630869+0000","last_active":"2024-03-30T07:45:00.624731+0000","last_peered":"2024-03-30T07:45:00.624731+0000","last_clean":"2024-03-30T07:45:00.624731+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:45:00.624731+0000","last_undegraded":"2024-03-30T07:45:00.624731+0000","last_fullsized":"2024-03-30T07:45:00.624731+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"994'8","last_scrub_stamp":"2024-03-30T07:45:00.623836+0000","last_deep_scrub":"994'8","last_deep_scrub_stamp":"2024-03-30T07:45:00.623836+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.623836+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:48:12.552105+0000","scrub_duration":0.150706335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":948,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.223038+0000","last_change":"2024-03-30T07:45:04.228572+0000","last_active":"2024-03-30T07:45:04.223038+0000","last_peered":"2024-03-30T07:45:04.223038+0000","last_clean":"2024-03-30T07:45:04.223038+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:45:04.223038+0000","last_undegraded":"2024-03-30T07:45:04.223038+0000","last_fullsized":"2024-03-30T07:45:04.223038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.222263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.222263+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.222263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:27:33.606081+0000","scrub_duration":0.066534888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280862,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64652,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307188,"statfs":{"total":96636764160,"available":96570560512,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:25 2024","interfaces":[{"interface":"back","average":{"1min":9.173,"5min":18.387,"15min":18.98},"min":{"1min":5.0540000000000003,"5min":4.7649999999999997,"15min":4.7649999999999997},"max":{"1min":12.066000000000001,"5min":249.63200000000001,"15min":292.04199999999997},"last":8.9019999999999992},{"interface":"front","average":{"1min":12.994,"5min":21.666,"15min":20.100000000000001},"min":{"1min":6.0720000000000001,"5min":5.0730000000000004,"15min":4.79},"max":{"1min":39.451999999999998,"5min":248.767,"15min":299.35000000000002},"last":13.803000000000001}]},{"osd":1,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":11.307,"5min":13.699,"15min":17.414000000000001},"min":{"1min":6.0860000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":24.446999999999999,"5min":104.782,"15min":169.17099999999999},"last":8.1170000000000009},{"interface":"front","average":{"1min":16.443999999999999,"5min":19.584,"15min":24.741},"min":{"1min":7.851,"5min":7.851,"15min":6.7789999999999999},"max":{"1min":60.030999999999999,"5min":181.46199999999999,"15min":286.52199999999999},"last":15.285}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":1180.998,"5min":530.18200000000002,"15min":228.39699999999999},"min":{"1min":9.0640000000000001,"5min":9.0640000000000001,"15min":8.0619999999999994},"max":{"1min":14020.602000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":19.309000000000001},{"interface":"front","average":{"1min":1157.28,"5min":537.84500000000003,"15min":233.36799999999999},"min":{"1min":12.426,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13714.877,"5min":13714.877,"15min":13714.877},"last":18.509}]},{"osd":4,"last update":"Sat Mar 30 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":1421.798,"5min":613.84400000000005,"15min":308.387},"min":{"1min":7.923,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":12463.806,"5min":12556.546,"15min":12556.546},"last":10.787000000000001},{"interface":"front","average":{"1min":1426.5699999999999,"5min":619.529,"15min":314.673},"min":{"1min":12.487,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":12465.362999999999,"5min":12557.409,"15min":12557.409},"last":12.353}]},{"osd":5,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":27.942,"5min":27.231999999999999,"15min":51.024000000000001},"min":{"1min":9.3350000000000009,"5min":9.3350000000000009,"15min":9.3350000000000009},"max":{"1min":161.35400000000001,"5min":254.04300000000001,"15min":4768.4960000000001},"last":17.582000000000001},{"interface":"front","average":{"1min":32.487000000000002,"5min":32.831000000000003,"15min":51.594999999999999},"min":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":7.8760000000000003},"max":{"1min":165.96600000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":20.111999999999998}]},{"osd":6,"last update":"Sat Mar 30 07:42:20 2024","interfaces":[{"interface":"back","average":{"1min":57.975000000000001,"5min":40.822000000000003,"15min":28.888000000000002},"min":{"1min":14.178000000000001,"5min":12.103,"15min":6.9530000000000003},"max":{"1min":328.01600000000002,"5min":328.01600000000002,"15min":350.94400000000002},"last":10.173},{"interface":"front","average":{"1min":32.357999999999997,"5min":32.185000000000002,"15min":29.07},"min":{"1min":9.5039999999999996,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":98.340999999999994,"5min":294.49700000000001,"15min":323.77999999999997},"last":16.760999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:45:08 2024","interfaces":[{"interface":"back","average":{"1min":22.399000000000001,"5min":21.151,"15min":337.69299999999998},"min":{"1min":7.9219999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":179.583,"5min":334.75400000000002,"15min":19293.749},"last":11.500999999999999},{"interface":"front","average":{"1min":27.184999999999999,"5min":23.873999999999999,"15min":340.09699999999998},"min":{"1min":10.199,"5min":9.1430000000000007,"15min":8.7059999999999995},"max":{"1min":194.93299999999999,"5min":205.19800000000001,"15min":19292.978999999999},"last":15.573}]}]},{"osd":0,"up_from":22,"seq":94489280863,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":5.9450000000000003},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":17.783999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":11.815},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":14.210000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":1499.876,"5min":472.52499999999998,"15min":195.184},"min":{"1min":9.7899999999999991,"5min":9.7899999999999991,"15min":6.6429999999999998},"max":{"1min":13546.239,"5min":13546.239,"15min":13546.239},"last":16.184000000000001},{"interface":"front","average":{"1min":1557.163,"5min":474.86099999999999,"15min":194.38},"min":{"1min":7.4000000000000004,"5min":6.7530000000000001,"15min":5.2569999999999997},"max":{"1min":13851.101000000001,"5min":13851.101000000001,"15min":13851.101000000001},"last":10.968999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":22.648},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":18.710000000000001}]},{"osd":5,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":41.381999999999998,"5min":25.181000000000001,"15min":60.106000000000002},"min":{"1min":7.4539999999999997,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":460.38799999999998,"5min":460.38799999999998,"15min":7478.7280000000001},"last":12.727},{"interface":"front","average":{"1min":47.198,"5min":34.844999999999999,"15min":61.795000000000002},"min":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":7.125},"max":{"1min":470.99599999999998,"5min":470.99599999999998,"15min":7477.2579999999998},"last":23.946999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":44.006999999999998,"5min":37.201999999999998,"15min":29.158999999999999},"min":{"1min":13.180999999999999,"5min":12.316000000000001,"15min":8.7569999999999997},"max":{"1min":386.452,"5min":386.452,"15min":386.452},"last":19.843},{"interface":"front","average":{"1min":31.984999999999999,"5min":28.108000000000001,"15min":25.620999999999999},"min":{"1min":8.2970000000000006,"5min":7.835,"15min":7.835},"max":{"1min":300.58199999999999,"5min":300.58199999999999,"15min":300.58199999999999},"last":12.698}]},{"osd":7,"last update":"Sat Mar 30 07:45:07 2024","interfaces":[{"interface":"back","average":{"1min":15.141999999999999,"5min":20.492999999999999,"15min":327.12400000000002},"min":{"1min":7.2869999999999999,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":38.997,"5min":295.38600000000002,"15min":21868.147000000001},"last":20.196000000000002},{"interface":"front","average":{"1min":21.547999999999998,"5min":23.123999999999999,"15min":330.87700000000001},"min":{"1min":11.052,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":66.225999999999999,"5min":302.11500000000001,"15min":21842.699000000001},"last":13.308}]}]},{"osd":4,"up_from":20,"seq":85899346257,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78604,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293236,"statfs":{"total":96636764160,"available":96556273664,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":42.113,"5min":30.841000000000001,"15min":21.015000000000001},"min":{"1min":5.0019999999999998,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":193.56299999999999,"5min":350.36599999999999,"15min":350.36599999999999},"last":9.1690000000000005},{"interface":"front","average":{"1min":39.423000000000002,"5min":29.962,"15min":26.541},"min":{"1min":6.9080000000000004,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":196.321,"5min":410.93099999999998,"15min":410.93099999999998},"last":15.637}]},{"osd":1,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":26.478999999999999,"5min":24.071999999999999,"15min":20.161999999999999},"min":{"1min":5.5709999999999997,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":137.941,"5min":313.54399999999998,"15min":313.54399999999998},"last":7.7030000000000003},{"interface":"front","average":{"1min":33.582000000000001,"5min":33.255000000000003,"15min":26.905999999999999},"min":{"1min":8.0679999999999996,"5min":8.0679999999999996,"15min":6.3719999999999999},"max":{"1min":138.87200000000001,"5min":466.54599999999999,"15min":466.54599999999999},"last":17.655000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":36.649000000000001,"5min":24.920999999999999,"15min":25.821999999999999},"min":{"1min":7.6799999999999997,"5min":6.1139999999999999,"15min":6.1139999999999999},"max":{"1min":198.03100000000001,"5min":371.74200000000002,"15min":768.79100000000005},"last":11.212999999999999},{"interface":"front","average":{"1min":41.881,"5min":29.448,"15min":25.213000000000001},"min":{"1min":9.2889999999999997,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":200.797,"5min":372.322,"15min":372.322},"last":12.532999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":3380.04,"5min":1086.2429999999999,"15min":410.63499999999999},"min":{"1min":9.2530000000000001,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":13684.860000000001,"5min":13684.860000000001,"15min":13684.860000000001},"last":13.218999999999999},{"interface":"front","average":{"1min":3346.288,"5min":1072.2090000000001,"15min":407.68000000000001},"min":{"1min":10.25,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":13604.335999999999,"5min":13604.335999999999,"15min":13604.335999999999},"last":11.888999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":34.985999999999997,"5min":31.134,"15min":43.316000000000003},"min":{"1min":11.901999999999999,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":202.54400000000001,"5min":421.40600000000001,"15min":4807.1080000000002},"last":14.885999999999999},{"interface":"front","average":{"1min":32.462000000000003,"5min":31.606000000000002,"15min":46.908000000000001},"min":{"1min":9.0909999999999993,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":140.81399999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":10.462}]},{"osd":6,"last update":"Sat Mar 30 07:41:38 2024","interfaces":[{"interface":"back","average":{"1min":35.414000000000001,"5min":31.486000000000001,"15min":30.466999999999999},"min":{"1min":9.8079999999999998,"5min":9.4640000000000004,"15min":9.4640000000000004},"max":{"1min":223.85499999999999,"5min":529.13699999999994,"15min":529.13699999999994},"last":19.373999999999999},{"interface":"front","average":{"1min":38.140999999999998,"5min":34.231999999999999,"15min":36.503999999999998},"min":{"1min":13.925000000000001,"5min":11.499000000000001,"15min":7.1870000000000003},"max":{"1min":223.16999999999999,"5min":529.73000000000002,"15min":967.58399999999995},"last":16.373000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":41.027999999999999,"5min":36.396999999999998,"15min":341.108},"min":{"1min":11.250999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":219.53100000000001,"5min":530.32500000000005,"15min":20845.164000000001},"last":20.245999999999999},{"interface":"front","average":{"1min":39.600999999999999,"5min":46.209000000000003,"15min":345.48099999999999},"min":{"1min":9.8770000000000007,"5min":8.9100000000000001,"15min":7.5060000000000002},"max":{"1min":190.62799999999999,"5min":664.02300000000002,"15min":20888.814999999999},"last":18.288}]}]},{"osd":7,"up_from":19,"seq":81604378957,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":10.182,"5min":18.259,"15min":21.504000000000001},"min":{"1min":4.8689999999999998,"5min":4.8689999999999998,"15min":4.2789999999999999},"max":{"1min":14.627000000000001,"5min":358.94499999999999,"15min":368.16899999999998},"last":12.885},{"interface":"front","average":{"1min":9.3840000000000003,"5min":17.870999999999999,"15min":24.042999999999999},"min":{"1min":5.7149999999999999,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":23.771999999999998,"5min":343.65699999999998,"15min":349.97199999999998},"last":11.984999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":9.157,"5min":12.548999999999999,"15min":23.257000000000001},"min":{"1min":4.8639999999999999,"5min":4.577,"15min":4.577},"max":{"1min":18.594000000000001,"5min":84.540000000000006,"15min":472.33300000000003},"last":6.96},{"interface":"front","average":{"1min":13.550000000000001,"5min":18.526,"15min":25.012},"min":{"1min":9.4909999999999997,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":21.835999999999999,"5min":184.33500000000001,"15min":292.92099999999999},"last":13.558999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:45:04 2024","interfaces":[{"interface":"back","average":{"1min":13.602,"5min":23.742000000000001,"15min":31.006},"min":{"1min":8.6899999999999995,"5min":8.4410000000000007,"15min":7.1180000000000003},"max":{"1min":18.978999999999999,"5min":341.96100000000001,"15min":400.43099999999998},"last":14.965999999999999},{"interface":"front","average":{"1min":20.641999999999999,"5min":30.042999999999999,"15min":34.962000000000003},"min":{"1min":12.734,"5min":10.554,"15min":8.0380000000000003},"max":{"1min":103.166,"5min":389.37200000000001,"15min":439.31999999999999},"last":16.681000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":896.39800000000002,"5min":450.06999999999999,"15min":204.417},"min":{"1min":11.375999999999999,"5min":11.375999999999999,"15min":7.5940000000000003},"max":{"1min":13240.941000000001,"5min":13240.941000000001,"15min":13240.941000000001},"last":17.686},{"interface":"front","average":{"1min":915.66300000000001,"5min":450.815,"15min":204.88300000000001},"min":{"1min":7.484,"5min":7.423,"15min":7.423},"max":{"1min":13596.368,"5min":13596.368,"15min":13596.368},"last":8.6229999999999993}]},{"osd":4,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":1875.7170000000001,"5min":716.80100000000004,"15min":291.15100000000001},"min":{"1min":11.651999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":14214.021000000001,"5min":14214.021000000001,"15min":14214.021000000001},"last":18.542999999999999},{"interface":"front","average":{"1min":1866.454,"5min":725.48099999999999,"15min":295.13099999999997},"min":{"1min":6.1879999999999997,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":14206.152,"5min":14206.152,"15min":14206.152},"last":9.6460000000000008}]},{"osd":5,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":22.273,"5min":25.465,"15min":108.16500000000001},"min":{"1min":12.196999999999999,"5min":8.0340000000000007,"15min":7.4050000000000002},"max":{"1min":105.785,"5min":345.90499999999997,"15min":7694.1369999999997},"last":19.388000000000002},{"interface":"front","average":{"1min":19.018000000000001,"5min":20.571000000000002,"15min":112.063},"min":{"1min":8.3079999999999998,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":104.148,"5min":203.31299999999999,"15min":7725.4070000000002},"last":11.367000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:42:11 2024","interfaces":[{"interface":"back","average":{"1min":27.544,"5min":35.518999999999998,"15min":31.760999999999999},"min":{"1min":9.8190000000000008,"5min":8.7620000000000005,"15min":8.7620000000000005},"max":{"1min":171.96100000000001,"5min":316.012,"15min":316.012},"last":20.321999999999999},{"interface":"front","average":{"1min":35.613999999999997,"5min":36.930999999999997,"15min":29.056999999999999},"min":{"1min":18.420999999999999,"5min":6.8700000000000001,"15min":6.3090000000000002},"max":{"1min":194.72399999999999,"5min":743.27700000000004,"15min":743.27700000000004},"last":14.252000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477083,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60952,"kb_used_data":7696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310888,"statfs":{"total":96636764160,"available":96574349312,"internally_reserved":0,"allocated":7880704,"data_stored":4551427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":13.853999999999999,"5min":12.869,"15min":18.533999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":55.762,"5min":132.02699999999999,"15min":421.78699999999998},"last":13.811999999999999},{"interface":"front","average":{"1min":12.318,"5min":12.644,"15min":18.782},"min":{"1min":7.5750000000000002,"5min":5.2759999999999998,"15min":5.2009999999999996},"max":{"1min":40.883000000000003,"5min":78.022999999999996,"15min":409.91000000000003},"last":7.5439999999999996}]},{"osd":2,"last update":"Sat Mar 30 07:45:16 2024","interfaces":[{"interface":"back","average":{"1min":10.548,"5min":13.284000000000001,"15min":21.007000000000001},"min":{"1min":5.8360000000000003,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":30.565000000000001,"5min":180.42599999999999,"15min":433.34899999999999},"last":6.8209999999999997},{"interface":"front","average":{"1min":14.967000000000001,"5min":17.143000000000001,"15min":23.596},"min":{"1min":8.5860000000000003,"5min":8.1790000000000003,"15min":5.6619999999999999},"max":{"1min":26.742999999999999,"5min":177.982,"15min":624.27300000000002},"last":15.377000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1671.011,"5min":557.96100000000001,"15min":225.822},"min":{"1min":9.8309999999999995,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":12913.004000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":14.621},{"interface":"front","average":{"1min":1712.2249999999999,"5min":566.452,"15min":227.315},"min":{"1min":9.1899999999999995,"5min":7.1689999999999996,"15min":6.6130000000000004},"max":{"1min":13136.255999999999,"5min":13136.255999999999,"15min":13136.255999999999},"last":8.6349999999999998}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":16.131},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":9.2750000000000004}]},{"osd":5,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":16.731000000000002,"5min":19.794,"15min":46.238},"min":{"1min":9.8879999999999999,"5min":9.8879999999999999,"15min":8.3439999999999994},"max":{"1min":32.831000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":17.003},{"interface":"front","average":{"1min":15.535,"5min":17.545999999999999,"15min":44.427999999999997},"min":{"1min":10.175000000000001,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":22.318999999999999,"5min":123.297,"15min":7345.2740000000003},"last":9.8330000000000002}]},{"osd":6,"last update":"Sat Mar 30 07:45:17 2024","interfaces":[{"interface":"back","average":{"1min":21.728000000000002,"5min":21.728000000000002,"15min":21.728000000000002},"min":{"1min":11.571,"5min":11.571,"15min":11.571},"max":{"1min":73.358999999999995,"5min":73.358999999999995,"15min":73.358999999999995},"last":10.385},{"interface":"front","average":{"1min":24.260000000000002,"5min":24.260000000000002,"15min":24.260000000000002},"min":{"1min":7.6379999999999999,"5min":7.6379999999999999,"15min":7.6379999999999999},"max":{"1min":246.25200000000001,"5min":246.25200000000001,"15min":246.25200000000001},"last":11.744999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:45:16 2024","interfaces":[{"interface":"back","average":{"1min":19.524999999999999,"5min":16.539000000000001,"15min":242.172},"min":{"1min":9.8439999999999994,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":60.286000000000001,"5min":115.461,"15min":20279.343000000001},"last":12.334},{"interface":"front","average":{"1min":16.806999999999999,"5min":21.263999999999999,"15min":250.071},"min":{"1min":9.2230000000000008,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":58.329999999999998,"5min":281.423,"15min":20311.815999999999},"last":17.786000000000001}]}]},{"osd":5,"up_from":231,"seq":992137445703,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":14.800000000000001},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":8.7189999999999994}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":7.0369999999999999},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":16.968}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":16.143999999999998},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":9.3490000000000002}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":10.007999999999999},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":7.657}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":17.765000000000001},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":18.538}]},{"osd":6,"last update":"Sat Mar 30 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":16.585999999999999,"5min":16.585999999999999,"15min":16.585999999999999},"min":{"1min":6.9390000000000001,"5min":6.9390000000000001,"15min":6.9390000000000001},"max":{"1min":96.572999999999993,"5min":96.572999999999993,"15min":96.572999999999993},"last":12.965},{"interface":"front","average":{"1min":19.701000000000001,"5min":19.701000000000001,"15min":19.701000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":8.8109999999999999},"max":{"1min":97.343999999999994,"5min":97.343999999999994,"15min":97.343999999999994},"last":10.640000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":17.626999999999999,"5min":14.768000000000001,"15min":283.55200000000002},"min":{"1min":9.266,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":88.396000000000001,"5min":115.589,"15min":19723.973000000002},"last":14.084},{"interface":"front","average":{"1min":19.236000000000001,"5min":18.826000000000001,"15min":283.75799999999998},"min":{"1min":10.714,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":74.75,"5min":84.802000000000007,"15min":19724.860000000001},"last":13.382999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542483,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82700,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289140,"statfs":{"total":96636764160,"available":96552079360,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":12.987},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":10.01}]},{"osd":1,"last update":"Sat Mar 30 07:44:28 2024","interfaces":[{"interface":"back","average":{"1min":34.786999999999999,"5min":22.965,"15min":21.632999999999999},"min":{"1min":8.6310000000000002,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":291.73599999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":8.7789999999999999},{"interface":"front","average":{"1min":48.518999999999998,"5min":29.795999999999999,"15min":25.812000000000001},"min":{"1min":7.7089999999999996,"5min":7.5599999999999996,"15min":6.415},"max":{"1min":541.21199999999999,"5min":541.21199999999999,"15min":541.21199999999999},"last":9.1690000000000005}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":17.433},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":10.762}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":14.099},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":14.718999999999999}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":18.541},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":12.339}]},{"osd":6,"last update":"Sat Mar 30 07:42:36 2024","interfaces":[{"interface":"back","average":{"1min":21.367999999999999,"5min":31.001999999999999,"15min":27.120000000000001},"min":{"1min":11.406000000000001,"5min":8.3460000000000001,"15min":8.3460000000000001},"max":{"1min":110.607,"5min":435.96600000000001,"15min":435.96600000000001},"last":20.395},{"interface":"front","average":{"1min":21.904,"5min":24.468,"15min":25.545999999999999},"min":{"1min":10.824,"5min":10.608000000000001,"15min":8.1180000000000003},"max":{"1min":62.828000000000003,"5min":145.239,"15min":259.60399999999998},"last":16.68}]},{"osd":7,"last update":"Sat Mar 30 07:44:22 2024","interfaces":[{"interface":"back","average":{"1min":39.734000000000002,"5min":26.948,"15min":285.68700000000001},"min":{"1min":9.9580000000000002,"5min":9.9580000000000002,"15min":8.2690000000000001},"max":{"1min":527.00300000000004,"5min":527.00300000000004,"15min":20495.457999999999},"last":19.526},{"interface":"front","average":{"1min":30.413,"5min":28.954000000000001,"15min":287.50799999999998},"min":{"1min":11.025,"5min":11.025,"15min":11.025},"max":{"1min":308.75900000000001,"5min":308.75900000000001,"15min":20496.241000000002},"last":15.366}]}]},{"osd":6,"up_from":978,"seq":4200478015817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60560,"kb_used_data":7304,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311280,"statfs":{"total":96636764160,"available":96574750720,"internally_reserved":0,"allocated":7479296,"data_stored":4153587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":17.824000000000002,"5min":17.824000000000002,"15min":17.824000000000002},"min":{"1min":6.6159999999999997,"5min":6.6159999999999997,"15min":6.6159999999999997},"max":{"1min":125.791,"5min":125.791,"15min":125.791},"last":11.31},{"interface":"front","average":{"1min":26.898,"5min":26.898,"15min":26.898},"min":{"1min":7.5309999999999997,"5min":7.5309999999999997,"15min":7.5309999999999997},"max":{"1min":265.62900000000002,"5min":265.62900000000002,"15min":265.62900000000002},"last":10.356}]},{"osd":1,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":9.5139999999999993,"5min":9.5139999999999993,"15min":9.5139999999999993},"min":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"max":{"1min":20.934999999999999,"5min":20.934999999999999,"15min":20.934999999999999},"last":9.0619999999999994},{"interface":"front","average":{"1min":19.276,"5min":19.276,"15min":19.276},"min":{"1min":7.6959999999999997,"5min":7.6959999999999997,"15min":7.6959999999999997},"max":{"1min":128.578,"5min":128.578,"15min":128.578},"last":16.423999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":19.367999999999999,"5min":19.367999999999999,"15min":19.367999999999999},"min":{"1min":8.2680000000000007,"5min":8.2680000000000007,"15min":8.2680000000000007},"max":{"1min":119.96299999999999,"5min":119.96299999999999,"15min":119.96299999999999},"last":14.736000000000001},{"interface":"front","average":{"1min":22.373999999999999,"5min":22.373999999999999,"15min":22.373999999999999},"min":{"1min":11.938000000000001,"5min":11.938000000000001,"15min":11.938000000000001},"max":{"1min":124.18300000000001,"5min":124.18300000000001,"15min":124.18300000000001},"last":19.318000000000001}]},{"osd":3,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":1112.355,"5min":1112.355,"15min":1112.355},"min":{"1min":6.9489999999999998,"5min":6.9489999999999998,"15min":6.9489999999999998},"max":{"1min":7852.915,"5min":7852.915,"15min":7852.915},"last":7.8899999999999997},{"interface":"front","average":{"1min":1060.037,"5min":1060.037,"15min":1060.037},"min":{"1min":8.3620000000000001,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":7655.6350000000002,"5min":7655.6350000000002,"15min":7655.6350000000002},"last":13.885999999999999}]},{"osd":4,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":2263.2840000000001,"5min":2263.2840000000001,"15min":2263.2840000000001},"min":{"1min":10.058999999999999,"5min":10.058999999999999,"15min":10.058999999999999},"max":{"1min":14228.966,"5min":14228.966,"15min":14228.966},"last":17.343},{"interface":"front","average":{"1min":2260.0839999999998,"5min":2260.0839999999998,"15min":2260.0839999999998},"min":{"1min":6.1109999999999998,"5min":6.1109999999999998,"15min":6.1109999999999998},"max":{"1min":14232.103999999999,"5min":14232.103999999999,"15min":14232.103999999999},"last":12.089}]},{"osd":5,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":21.741,"5min":21.741,"15min":21.741},"min":{"1min":11.446,"5min":11.446,"15min":11.446},"max":{"1min":125.30800000000001,"5min":125.30800000000001,"15min":125.30800000000001},"last":18.384},{"interface":"front","average":{"1min":17.327999999999999,"5min":17.327999999999999,"15min":17.327999999999999},"min":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"max":{"1min":105.018,"5min":105.018,"15min":105.018},"last":12.714}]},{"osd":7,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":31.873000000000001,"5min":31.873000000000001,"15min":31.873000000000001},"min":{"1min":11.654999999999999,"5min":11.654999999999999,"15min":11.654999999999999},"max":{"1min":246.78100000000001,"5min":246.78100000000001,"15min":246.78100000000001},"last":20.218},{"interface":"front","average":{"1min":22.234000000000002,"5min":22.234000000000002,"15min":22.234000000000002},"min":{"1min":10.59,"5min":10.59,"15min":10.59},"max":{"1min":107.79300000000001,"5min":107.79300000000001,"15min":107.79300000000001},"last":13.295}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:45:21.537 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:21.537 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:21.538 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-03-30T07:17:52.426377+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=7, tm_min=44, tm_sec=52, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T07:45:21.538 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-30T07:45:22.311 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:50.121 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.312 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:50.123 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.324 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:29:50.135 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.357 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:50.168 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.362 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:50.139 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.362 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:50.173 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.365 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:50.142 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.373 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:29:50.184 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:22.375 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:29:50.152 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.355 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:05.165 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.356 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:05.167 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.372 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:05.183 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.410 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:05.220 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.410 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:05.187 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.411 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:05.222 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.412 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:05.189 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.426 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:05.237 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:37.427 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:05.204 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:41.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T07:45:41.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.641+0000 7f1451c84640 10 monclient: get_monmap_and_config 2024-03-30T07:45:41.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.641+0000 7f1451c84640 10 monclient: build_initial_monmap 2024-03-30T07:45:41.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.641+0000 7f1451c84640 10 monclient: monmap: 2024-03-30T07:45:41.643 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:45:41.642280+0000 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:45:41.642280+0000 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:45:41.644 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 Processor -- start 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 -- start start 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient: init 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-b con 0x7f144c0adad0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f144c0adeb0 0x7f144c071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-c con 0x7f144c0adeb0 addr v2:172.21.15.52:3300/0 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f144c06e550 0x7f144c071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-a con 0x7f144c06e550 addr v2:172.21.15.47:3300/0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c13e310 con 0x7f144c06e550 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c06eb90 con 0x7f144c0adeb0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c071b90 con 0x7f144c0adad0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.642+0000 7f1451c84640 10 monclient(hunting): authenticate will time out at 3267.758022s 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f144c06e550 0x7f144c071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144affd640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f144c0adeb0 0x7f144c071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f144c06e550 0x7f144c071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54498/0 (socket says 172.21.15.47:54498) 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60292/0 (socket says 172.21.15.47:60292) 2024-03-30T07:45:41.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 1 -- 172.21.15.47:0/1150035727 learned_addr learned my addr 172.21.15.47:0/1150035727 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 10 monclient(hunting): get_auth_request con 0x7f144c06e550 auth_method 0 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 10 monclient(hunting): get_auth_request con 0x7f144c0adad0 auth_method 0 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.643+0000 7f144b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): get_auth_request con 0x7f144c0adeb0 auth_method 0 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.644+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 10 monclient(hunting): handle_auth_done global_id 15936 payload 274 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 -- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 msgr2=0x7f144c13ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 -- 172.21.15.47:0/1150035727 >> v2:172.21.15.52:3300/0 conn(0x7f144c0adeb0 msgr2=0x7f144c071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 --2- 172.21.15.47:0/1150035727 >> v2:172.21.15.52:3300/0 conn(0x7f144c0adeb0 0x7f144c071110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 10 monclient: _finish_hunting 0 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 monclient: found mon.noname-a 2024-03-30T07:45:41.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:41.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:45:41.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144bfff640 1 -- 172.21.15.47:0/1150035727 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c070a10 con 0x7f144c06e550 2024-03-30T07:45:41.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.646+0000 7f144b7fe640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T07:45:41.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.648+0000 7f144bfff640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c071650 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f144c072080 tx=0x7f144402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4a40b54c94040c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 1 -- 172.21.15.47:0/1150035727 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144402c950 con 0x7f144c06e550 2024-03-30T07:45:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:45:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: dump: 2024-03-30T07:45:41.654 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: _finish_auth 0 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: _check_auth_tickets 2024-03-30T07:45:41.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 1 -- 172.21.15.47:0/1150035727 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144402cab0 con 0x7f144c06e550 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 1 -- 172.21.15.47:0/1150035727 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144402ce40 con 0x7f144c06e550 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f144a7fc640 10 monclient: dump: 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.656 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f1451c84640 5 monclient: authenticate success, global_id 15936 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f1451c84640 10 monclient: get_monmap_and_config success 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.651+0000 7f1451c84640 4 set_mon_vals no callback set 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 10 monclient: shutdown 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 -- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 msgr2=0x7f144c071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c071650 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f144c072080 tx=0x7f144402bdb0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 -- 172.21.15.47:0/1150035727 shutdown_connections 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c071650 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 --2- 172.21.15.47:0/1150035727 >> v2:172.21.15.52:3300/0 conn(0x7f144c0adeb0 0x7f144c071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 --2- 172.21.15.47:0/1150035727 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c13ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 -- 172.21.15.47:0/1150035727 >> 172.21.15.47:0/1150035727 conn(0x7f144c0b30d0 msgr2=0x7f144c0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:41.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 -- 172.21.15.47:0/1150035727 shutdown_connections 2024-03-30T07:45:41.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.652+0000 7f1451c84640 1 -- 172.21.15.47:0/1150035727 wait complete. 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 10 monclient: build_initial_monmap 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 10 monclient: monmap: 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:41.661 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 librados: starting msgr at 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 librados: starting objecter 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 Processor -- start 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 -- start start 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 librados: setting wanted keys 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 1 librados: calling monclient init 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.653+0000 7f1451c84640 10 monclient: init 2024-03-30T07:45:41.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-a con 0x7f144c06e550 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c1d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-c con 0x7f144c0adad0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f144c136380 0x7f144c1d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): picked mon.noname-b con 0x7f144c136380 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c13a150 con 0x7f144c06e550 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c1425b0 con 0x7f144c136380 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c141a00 con 0x7f144c0adad0 2024-03-30T07:45:41.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f1451c84640 10 monclient(hunting): authenticate will time out at 3267.769507s 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f144affd640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c1d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.654+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f144c136380 0x7f144c1d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:54510/0 (socket says 172.21.15.47:54510) 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 1 -- 172.21.15.47:0/3177376115 learned_addr learned my addr 172.21.15.47:0/3177376115 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): get_auth_request con 0x7f144c06e550 auth_method 0 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): get_auth_request con 0x7f144c0adad0 auth_method 0 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): get_auth_request con 0x7f144c136380 auth_method 0 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:41.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.655+0000 7f144bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 10 monclient(hunting): handle_auth_done global_id 15939 payload 824 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 -- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 msgr2=0x7f144c1d74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c1d74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 -- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f144c136380 msgr2=0x7f144c1d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f144c136380 0x7f144c1d7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 10 monclient: _finish_hunting 0 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 monclient: found mon.noname-a 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.47:3300/0 2024-03-30T07:45:41.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.658+0000 7f144b7fe640 1 -- 172.21.15.47:0/3177376115 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143c009400 con 0x7f144c06e550 2024-03-30T07:45:41.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.660+0000 7f144b7fe640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e8e0 tx=0x7f143c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:41.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.660+0000 7f1433fff640 10 client.?.objecter ms_handle_connect 0x7f144c06e550 2024-03-30T07:45:41.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.660+0000 7f1433fff640 10 client.?.objecter resend_mon_ops 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00c940 con 0x7f144c06e550 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: dump: 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: _finish_auth 0 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: _check_auth_tickets 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143c00caa0 con 0x7f144c06e550 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 5 monclient: authenticate success, global_id 15939 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 10 monclient: _renew_subs 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1444009b30 con 0x7f144c06e550 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00ce30 con 0x7f144c06e550 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:41.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1433fff640 10 monclient: dump: 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1449ffb640 4 set_mon_vals no callback set 2024-03-30T07:45:41.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 10 client.15939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 10 monclient: _renew_subs 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c13f270 con 0x7f144c06e550 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.664+0000 7f1451c84640 1 librados: init done 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.665+0000 7f1451c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.665+0000 7f1451c84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.665+0000 7f1451c84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.47:3300/0 2024-03-30T07:45:41.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.665+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c139f10 con 0x7f144c06e550 2024-03-30T07:45:41.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.683+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f143c013470 con 0x7f144c06e550 2024-03-30T07:45:41.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.683+0000 7f1433fff640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 0x7f143806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:41.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.684+0000 7f144affd640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 0x7f143806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:41.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.684+0000 7f144affd640 10 monclient: get_auth_request con 0x7f143806c610 auth_method 0 2024-03-30T07:45:41.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.685+0000 7f1433fff640 10 client.15939.objecter ms_handle_connect 0x7f143806c610 2024-03-30T07:45:41.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.685+0000 7f144affd640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 0x7f143806ead0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f144c0b0030 tx=0x7f1440009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:41.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.686+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f143c058c80 con 0x7f144c06e550 2024-03-30T07:45:41.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.686+0000 7f1433fff640 10 client.15939.objecter ms_dispatch 0x7f144c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:45:41.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.686+0000 7f1433fff640 3 client.15939.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:45:41.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.686+0000 7f1433fff640 3 client.15939.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:45:41.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.686+0000 7f1433fff640 20 client.15939.objecter dump_active .. 0 homeless 2024-03-30T07:45:41.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.942+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f143c054fc0 con 0x7f144c06e550 2024-03-30T07:45:41.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.942+0000 7f1433fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:41.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:41.942+0000 7f1433fff640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:45:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.128+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 --> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f144c13f0f0 con 0x7f143806c610 2024-03-30T07:45:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.134+0000 7f1433fff640 1 -- 172.21.15.47:0/3177376115 <== mgr.4100 v2:172.21.15.52:6800/34694 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152359 (secure 0 0 0) 0x7f1440011a30 con 0x7f143806c610 2024-03-30T07:45:42.138 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-03-30T07:45:42.139 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.136+0000 7f1451c84640 10 librados: watch_flush enter 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.136+0000 7f1451c84640 10 librados: watch_flush exit 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.136+0000 7f1451c84640 20 client.15939.objecter shutdown clearing up homeless session... 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.136+0000 7f1451c84640 10 client.15939.objecter successfully canceled tick 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 msgr2=0x7f143806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 0x7f143806ead0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f144c0b0030 tx=0x7f1440009190 comp rx=0 tx=0).stop 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 10 monclient: shutdown 2024-03-30T07:45:42.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 msgr2=0x7f144c1399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e8e0 tx=0x7f143c00eda0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 shutdown_connections 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f143806c610 0x7f143806ead0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f144c136380 0x7f144c1d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f144c0adad0 0x7f144c1d74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 --2- 172.21.15.47:0/3177376115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f144c06e550 0x7f144c1399d0 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 >> 172.21.15.47:0/3177376115 conn(0x7f144c0b30d0 msgr2=0x7f144c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 shutdown_connections 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 -- 172.21.15.47:0/3177376115 wait complete. 2024-03-30T07:45:42.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.137+0000 7f1451c84640 1 librados: shutdown 2024-03-30T07:45:42.150 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1873,"stamp":"2024-03-30T07:45:40.382564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":529528,"kb_used_data":59192,"kb_used_omap":12,"kb_used_meta":470259,"kb_avail":754445192,"statfs":{"total":773094113280,"available":772551876608,"internally_reserved":0,"allocated":60612608,"data_stored":34024346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":481545784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"16.c","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.100288+0000","last_change":"2024-03-30T07:45:03.102050+0000","last_active":"2024-03-30T07:45:03.100288+0000","last_peered":"2024-03-30T07:45:03.100288+0000","last_clean":"2024-03-30T07:45:03.100288+0000","last_became_active":"2024-03-30T07:18:12.748627+0000","last_became_peered":"2024-03-30T07:18:12.748627+0000","last_unstale":"2024-03-30T07:45:03.100288+0000","last_undegraded":"2024-03-30T07:45:03.100288+0000","last_fullsized":"2024-03-30T07:45:03.100288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.099490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.099490+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.099490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:00:52.186954+0000","scrub_duration":0.044015481000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":958,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.136050+0000","last_change":"2024-03-30T07:45:04.137799+0000","last_active":"2024-03-30T07:45:04.136050+0000","last_peered":"2024-03-30T07:45:04.136050+0000","last_clean":"2024-03-30T07:45:04.136050+0000","last_became_active":"2024-03-30T07:24:14.837417+0000","last_became_peered":"2024-03-30T07:24:14.837417+0000","last_unstale":"2024-03-30T07:45:04.136050+0000","last_undegraded":"2024-03-30T07:45:04.136050+0000","last_fullsized":"2024-03-30T07:45:04.136050+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.135262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.135262+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.135262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:11:25.177760+0000","scrub_duration":0.075101082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":933,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:11.563546+0000","last_change":"2024-03-30T07:45:11.565294+0000","last_active":"2024-03-30T07:45:11.563546+0000","last_peered":"2024-03-30T07:45:11.563546+0000","last_clean":"2024-03-30T07:45:11.563546+0000","last_became_active":"2024-03-30T07:18:07.987274+0000","last_became_peered":"2024-03-30T07:18:07.987274+0000","last_unstale":"2024-03-30T07:45:11.563546+0000","last_undegraded":"2024-03-30T07:45:11.563546+0000","last_fullsized":"2024-03-30T07:45:11.563546+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:11.562727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:11.562727+0000","last_clean_scrub_stamp":"2024-03-30T07:45:11.562727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:13:02.794032+0000","scrub_duration":0.033870826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:05.286114+0000","last_change":"2024-03-30T07:45:05.286114+0000","last_active":"2024-03-30T07:45:05.286114+0000","last_peered":"2024-03-30T07:45:05.286114+0000","last_clean":"2024-03-30T07:45:05.286114+0000","last_became_active":"2024-03-30T07:24:15.821549+0000","last_became_peered":"2024-03-30T07:24:15.821549+0000","last_unstale":"2024-03-30T07:45:05.286114+0000","last_undegraded":"2024-03-30T07:45:05.286114+0000","last_fullsized":"2024-03-30T07:45:05.286114+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:05.279442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:05.279442+0000","last_clean_scrub_stamp":"2024-03-30T07:45:05.279442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:51:04.595437+0000","scrub_duration":0.073014201000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":925,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.857888+0000","last_change":"2024-03-30T07:45:03.859580+0000","last_active":"2024-03-30T07:45:03.857888+0000","last_peered":"2024-03-30T07:45:03.857888+0000","last_clean":"2024-03-30T07:45:03.857888+0000","last_became_active":"2024-03-30T07:18:07.651920+0000","last_became_peered":"2024-03-30T07:18:07.651920+0000","last_unstale":"2024-03-30T07:45:03.857888+0000","last_undegraded":"2024-03-30T07:45:03.857888+0000","last_fullsized":"2024-03-30T07:45:03.857888+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.857105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.857105+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.857105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:45.010470+0000","scrub_duration":0.034940751999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":936,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.891505+0000","last_change":"2024-03-30T07:45:04.893251+0000","last_active":"2024-03-30T07:45:04.891505+0000","last_peered":"2024-03-30T07:45:04.891505+0000","last_clean":"2024-03-30T07:45:04.891505+0000","last_became_active":"2024-03-30T07:44:18.241138+0000","last_became_peered":"2024-03-30T07:44:18.241138+0000","last_unstale":"2024-03-30T07:45:04.891505+0000","last_undegraded":"2024-03-30T07:45:04.891505+0000","last_fullsized":"2024-03-30T07:45:04.891505+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.890711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.890711+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.890711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:43:45.934457+0000","scrub_duration":0.051333487999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":949,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.574165+0000","last_change":"2024-03-30T07:45:03.575953+0000","last_active":"2024-03-30T07:45:03.574165+0000","last_peered":"2024-03-30T07:45:03.574165+0000","last_clean":"2024-03-30T07:45:03.574165+0000","last_became_active":"2024-03-30T07:44:23.326694+0000","last_became_peered":"2024-03-30T07:44:23.326694+0000","last_unstale":"2024-03-30T07:45:03.574165+0000","last_undegraded":"2024-03-30T07:45:03.574165+0000","last_fullsized":"2024-03-30T07:45:03.574165+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":980,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.573356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.573356+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.573356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:59:20.825982+0000","scrub_duration":0.047055578000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":935,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:07.546857+0000","last_change":"2024-03-30T07:45:07.548601+0000","last_active":"2024-03-30T07:45:07.546857+0000","last_peered":"2024-03-30T07:45:07.546857+0000","last_clean":"2024-03-30T07:45:07.546857+0000","last_became_active":"2024-03-30T07:17:58.916735+0000","last_became_peered":"2024-03-30T07:17:58.916735+0000","last_unstale":"2024-03-30T07:45:07.546857+0000","last_undegraded":"2024-03-30T07:45:07.546857+0000","last_fullsized":"2024-03-30T07:45:07.546857+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:07.546069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:07.546069+0000","last_clean_scrub_stamp":"2024-03-30T07:45:07.546069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:17:25.702685+0000","scrub_duration":0.073536952000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":938,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.961851+0000","last_change":"2024-03-30T07:45:00.961851+0000","last_active":"2024-03-30T07:45:00.961851+0000","last_peered":"2024-03-30T07:45:00.961851+0000","last_clean":"2024-03-30T07:45:00.961851+0000","last_became_active":"2024-03-30T07:24:15.690808+0000","last_became_peered":"2024-03-30T07:24:15.690808+0000","last_unstale":"2024-03-30T07:45:00.961851+0000","last_undegraded":"2024-03-30T07:45:00.961851+0000","last_fullsized":"2024-03-30T07:45:00.961851+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:00.959383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:00.959383+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.959383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:36:05.184291+0000","scrub_duration":0.042599705000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":924,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:59.936597+0000","last_change":"2024-03-30T07:44:59.936597+0000","last_active":"2024-03-30T07:44:59.936597+0000","last_peered":"2024-03-30T07:44:59.936597+0000","last_clean":"2024-03-30T07:44:59.936597+0000","last_became_active":"2024-03-30T07:18:07.820544+0000","last_became_peered":"2024-03-30T07:18:07.820544+0000","last_unstale":"2024-03-30T07:44:59.936597+0000","last_undegraded":"2024-03-30T07:44:59.936597+0000","last_fullsized":"2024-03-30T07:44:59.936597+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:59.934115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:59.934115+0000","last_clean_scrub_stamp":"2024-03-30T07:44:59.934115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:42:05.163853+0000","scrub_duration":0.052884168000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":938,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:19.815365+0000","last_change":"2024-03-30T07:45:19.817079+0000","last_active":"2024-03-30T07:45:19.815365+0000","last_peered":"2024-03-30T07:45:19.815365+0000","last_clean":"2024-03-30T07:45:19.815365+0000","last_became_active":"2024-03-30T07:24:13.732254+0000","last_became_peered":"2024-03-30T07:24:13.732254+0000","last_unstale":"2024-03-30T07:45:19.815365+0000","last_undegraded":"2024-03-30T07:45:19.815365+0000","last_fullsized":"2024-03-30T07:45:19.815365+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:19.814563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:19.814563+0000","last_clean_scrub_stamp":"2024-03-30T07:45:19.814563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:32:34.521466+0000","scrub_duration":0.062398418999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.948801+0000","last_change":"2024-03-30T07:45:01.959948+0000","last_active":"2024-03-30T07:45:01.948801+0000","last_peered":"2024-03-30T07:45:01.948801+0000","last_clean":"2024-03-30T07:45:01.948801+0000","last_became_active":"2024-03-30T07:18:16.990644+0000","last_became_peered":"2024-03-30T07:18:16.990644+0000","last_unstale":"2024-03-30T07:45:01.948801+0000","last_undegraded":"2024-03-30T07:45:01.948801+0000","last_fullsized":"2024-03-30T07:45:01.948801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.948015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.948015+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.948015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:07:50.406960+0000","scrub_duration":0.075824290000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.762320+0000","last_change":"2024-03-30T07:45:01.769055+0000","last_active":"2024-03-30T07:45:01.762320+0000","last_peered":"2024-03-30T07:45:01.762320+0000","last_clean":"2024-03-30T07:45:01.762320+0000","last_became_active":"2024-03-30T07:16:49.932281+0000","last_became_peered":"2024-03-30T07:16:49.932281+0000","last_unstale":"2024-03-30T07:45:01.762320+0000","last_undegraded":"2024-03-30T07:45:01.762320+0000","last_fullsized":"2024-03-30T07:45:01.762320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.761515+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:12:32.025372+0000","scrub_duration":0.054478171999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.621808+0000","last_change":"2024-03-30T07:45:02.624117+0000","last_active":"2024-03-30T07:45:02.621808+0000","last_peered":"2024-03-30T07:45:02.621808+0000","last_clean":"2024-03-30T07:45:02.621808+0000","last_became_active":"2024-03-30T07:18:21.564489+0000","last_became_peered":"2024-03-30T07:18:21.564489+0000","last_unstale":"2024-03-30T07:45:02.621808+0000","last_undegraded":"2024-03-30T07:45:02.621808+0000","last_fullsized":"2024-03-30T07:45:02.621808+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.620958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.620958+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.620958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:47:25.436827+0000","scrub_duration":0.038212127999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":930,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:57.910639+0000","last_change":"2024-03-30T07:44:57.910639+0000","last_active":"2024-03-30T07:44:57.910639+0000","last_peered":"2024-03-30T07:44:57.910639+0000","last_clean":"2024-03-30T07:44:57.910639+0000","last_became_active":"2024-03-30T07:16:51.099768+0000","last_became_peered":"2024-03-30T07:16:51.099768+0000","last_unstale":"2024-03-30T07:44:57.910639+0000","last_undegraded":"2024-03-30T07:44:57.910639+0000","last_fullsized":"2024-03-30T07:44:57.910639+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:57.907974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:57.907974+0000","last_clean_scrub_stamp":"2024-03-30T07:44:57.907974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:10:30.858711+0000","scrub_duration":0.04023471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":923,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:05.848774+0000","last_change":"2024-03-30T07:45:05.850948+0000","last_active":"2024-03-30T07:45:05.848774+0000","last_peered":"2024-03-30T07:45:05.848774+0000","last_clean":"2024-03-30T07:45:05.848774+0000","last_became_active":"2024-03-30T07:18:16.044111+0000","last_became_peered":"2024-03-30T07:18:16.044111+0000","last_unstale":"2024-03-30T07:45:05.848774+0000","last_undegraded":"2024-03-30T07:45:05.848774+0000","last_fullsized":"2024-03-30T07:45:05.848774+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:05.847955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:05.847955+0000","last_clean_scrub_stamp":"2024-03-30T07:45:05.847955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:22:18.393244+0000","scrub_duration":0.031780224000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:08.737456+0000","last_change":"2024-03-30T07:45:08.743279+0000","last_active":"2024-03-30T07:45:08.737456+0000","last_peered":"2024-03-30T07:45:08.737456+0000","last_clean":"2024-03-30T07:45:08.737456+0000","last_became_active":"2024-03-30T07:16:49.694177+0000","last_became_peered":"2024-03-30T07:16:49.694177+0000","last_unstale":"2024-03-30T07:45:08.737456+0000","last_undegraded":"2024-03-30T07:45:08.737456+0000","last_fullsized":"2024-03-30T07:45:08.737456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:08.736649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:08.736649+0000","last_clean_scrub_stamp":"2024-03-30T07:45:08.736649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:28:46.083314+0000","scrub_duration":0.037436690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":922,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:11.966673+0000","last_change":"2024-03-30T07:45:11.968405+0000","last_active":"2024-03-30T07:45:11.966673+0000","last_peered":"2024-03-30T07:45:11.966673+0000","last_clean":"2024-03-30T07:45:11.966673+0000","last_became_active":"2024-03-30T07:18:16.865367+0000","last_became_peered":"2024-03-30T07:18:16.865367+0000","last_unstale":"2024-03-30T07:45:11.966673+0000","last_undegraded":"2024-03-30T07:45:11.966673+0000","last_fullsized":"2024-03-30T07:45:11.966673+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:11.965874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:11.965874+0000","last_clean_scrub_stamp":"2024-03-30T07:45:11.965874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:06:09.430315+0000","scrub_duration":0.080360671999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":965,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.226721+0000","last_change":"2024-03-30T07:45:02.228821+0000","last_active":"2024-03-30T07:45:02.226721+0000","last_peered":"2024-03-30T07:45:02.226721+0000","last_clean":"2024-03-30T07:45:02.226721+0000","last_became_active":"2024-03-30T07:44:18.301986+0000","last_became_peered":"2024-03-30T07:44:18.301986+0000","last_unstale":"2024-03-30T07:45:02.226721+0000","last_undegraded":"2024-03-30T07:45:02.226721+0000","last_fullsized":"2024-03-30T07:45:02.226721+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.225916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.225916+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.225916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:12:30.187116+0000","scrub_duration":0.055995208999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:03.578682+0000","last_change":"2024-03-30T07:45:03.580413+0000","last_active":"2024-03-30T07:45:03.578682+0000","last_peered":"2024-03-30T07:45:03.578682+0000","last_clean":"2024-03-30T07:45:03.578682+0000","last_became_active":"2024-03-30T07:17:58.336938+0000","last_became_peered":"2024-03-30T07:17:58.336938+0000","last_unstale":"2024-03-30T07:45:03.578682+0000","last_undegraded":"2024-03-30T07:45:03.578682+0000","last_fullsized":"2024-03-30T07:45:03.578682+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:03.577615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:03.577615+0000","last_clean_scrub_stamp":"2024-03-30T07:45:03.577615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T18:38:39.511514+0000","scrub_duration":0.034900265999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":946,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:15.644763+0000","last_change":"2024-03-30T07:45:15.646390+0000","last_active":"2024-03-30T07:45:15.644763+0000","last_peered":"2024-03-30T07:45:15.644763+0000","last_clean":"2024-03-30T07:45:15.644763+0000","last_became_active":"2024-03-30T07:44:23.543771+0000","last_became_peered":"2024-03-30T07:44:23.543771+0000","last_unstale":"2024-03-30T07:45:15.644763+0000","last_undegraded":"2024-03-30T07:45:15.644763+0000","last_fullsized":"2024-03-30T07:45:15.644763+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:15.643975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:15.643975+0000","last_clean_scrub_stamp":"2024-03-30T07:45:15.643975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:38:18.694937+0000","scrub_duration":0.031838125000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":940,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.469210+0000","last_change":"2024-03-30T07:45:01.470886+0000","last_active":"2024-03-30T07:45:01.469210+0000","last_peered":"2024-03-30T07:45:01.469210+0000","last_clean":"2024-03-30T07:45:01.469210+0000","last_became_active":"2024-03-30T07:44:18.155986+0000","last_became_peered":"2024-03-30T07:44:18.155986+0000","last_unstale":"2024-03-30T07:45:01.469210+0000","last_undegraded":"2024-03-30T07:45:01.469210+0000","last_fullsized":"2024-03-30T07:45:01.469210+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.468400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.468400+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.468400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T19:34:12.252367+0000","scrub_duration":0.029422416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":956,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.819965+0000","last_change":"2024-03-30T07:45:00.828787+0000","last_active":"2024-03-30T07:45:00.819965+0000","last_peered":"2024-03-30T07:45:00.819965+0000","last_clean":"2024-03-30T07:45:00.819965+0000","last_became_active":"2024-03-30T07:24:14.179605+0000","last_became_peered":"2024-03-30T07:24:14.179605+0000","last_unstale":"2024-03-30T07:45:00.819965+0000","last_undegraded":"2024-03-30T07:45:00.819965+0000","last_fullsized":"2024-03-30T07:45:00.819965+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:00.817708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:00.817708+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.817708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:21:03.691881+0000","scrub_duration":0.20702821599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":934,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:06.531324+0000","last_change":"2024-03-30T07:45:06.533065+0000","last_active":"2024-03-30T07:45:06.531324+0000","last_peered":"2024-03-30T07:45:06.531324+0000","last_clean":"2024-03-30T07:45:06.531324+0000","last_became_active":"2024-03-30T07:18:07.798509+0000","last_became_peered":"2024-03-30T07:18:07.798509+0000","last_unstale":"2024-03-30T07:45:06.531324+0000","last_undegraded":"2024-03-30T07:45:06.531324+0000","last_fullsized":"2024-03-30T07:45:06.531324+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:06.530509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:06.530509+0000","last_clean_scrub_stamp":"2024-03-30T07:45:06.530509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T17:44:12.438188+0000","scrub_duration":0.048993979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:57.719569+0000","last_change":"2024-03-30T07:44:57.719569+0000","last_active":"2024-03-30T07:44:57.719569+0000","last_peered":"2024-03-30T07:44:57.719569+0000","last_clean":"2024-03-30T07:44:57.719569+0000","last_became_active":"2024-03-30T07:24:13.383291+0000","last_became_peered":"2024-03-30T07:24:13.383291+0000","last_unstale":"2024-03-30T07:44:57.719569+0000","last_undegraded":"2024-03-30T07:44:57.719569+0000","last_fullsized":"2024-03-30T07:44:57.719569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:57.716513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:57.716513+0000","last_clean_scrub_stamp":"2024-03-30T07:44:57.716513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:14:29.808330+0000","scrub_duration":0.042252514999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":957,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:59.102794+0000","last_change":"2024-03-30T07:44:59.108606+0000","last_active":"2024-03-30T07:44:59.102794+0000","last_peered":"2024-03-30T07:44:59.102794+0000","last_clean":"2024-03-30T07:44:59.102794+0000","last_became_active":"2024-03-30T07:18:17.825710+0000","last_became_peered":"2024-03-30T07:18:17.825710+0000","last_unstale":"2024-03-30T07:44:59.102794+0000","last_undegraded":"2024-03-30T07:44:59.102794+0000","last_fullsized":"2024-03-30T07:44:59.102794+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:59.101982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:59.101982+0000","last_clean_scrub_stamp":"2024-03-30T07:44:59.101982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:03.681571+0000","scrub_duration":0.034080829,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":932,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:58.742553+0000","last_change":"2024-03-30T07:44:58.744347+0000","last_active":"2024-03-30T07:44:58.742553+0000","last_peered":"2024-03-30T07:44:58.742553+0000","last_clean":"2024-03-30T07:44:58.742553+0000","last_became_active":"2024-03-30T07:16:49.952118+0000","last_became_peered":"2024-03-30T07:16:49.952118+0000","last_unstale":"2024-03-30T07:44:58.742553+0000","last_undegraded":"2024-03-30T07:44:58.742553+0000","last_fullsized":"2024-03-30T07:44:58.742553+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:44:58.741755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:44:58.741755+0000","last_clean_scrub_stamp":"2024-03-30T07:44:58.741755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:27:46.255761+0000","scrub_duration":0.051197923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"994'78","reported_seq":1091,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:44:56.280200+0000","last_change":"2024-03-30T07:44:56.282541+0000","last_active":"2024-03-30T07:44:56.280200+0000","last_peered":"2024-03-30T07:44:56.280200+0000","last_clean":"2024-03-30T07:44:56.280200+0000","last_became_active":"2024-03-30T07:24:13.754440+0000","last_became_peered":"2024-03-30T07:24:13.754440+0000","last_unstale":"2024-03-30T07:44:56.280200+0000","last_undegraded":"2024-03-30T07:44:56.280200+0000","last_fullsized":"2024-03-30T07:44:56.280200+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"994'78","last_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_deep_scrub":"994'78","last_deep_scrub_stamp":"2024-03-30T07:44:56.279300+0000","last_clean_scrub_stamp":"2024-03-30T07:44:56.279300+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:20:04.428365+0000","scrub_duration":0.11938407600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":962,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.973792+0000","last_change":"2024-03-30T07:45:01.980356+0000","last_active":"2024-03-30T07:45:01.973792+0000","last_peered":"2024-03-30T07:45:01.973792+0000","last_clean":"2024-03-30T07:45:01.973792+0000","last_became_active":"2024-03-30T07:18:08.718592+0000","last_became_peered":"2024-03-30T07:18:08.718592+0000","last_unstale":"2024-03-30T07:45:01.973792+0000","last_undegraded":"2024-03-30T07:45:01.973792+0000","last_fullsized":"2024-03-30T07:45:01.973792+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.971705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.971705+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.971705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:57:44.662538+0000","scrub_duration":0.050853467999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":970,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.174685+0000","last_change":"2024-03-30T07:45:02.174685+0000","last_active":"2024-03-30T07:45:02.174685+0000","last_peered":"2024-03-30T07:45:02.174685+0000","last_clean":"2024-03-30T07:45:02.174685+0000","last_became_active":"2024-03-30T07:24:13.774622+0000","last_became_peered":"2024-03-30T07:24:13.774622+0000","last_unstale":"2024-03-30T07:45:02.174685+0000","last_undegraded":"2024-03-30T07:45:02.174685+0000","last_fullsized":"2024-03-30T07:45:02.174685+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.172128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.172128+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.172128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:32:49.128070+0000","scrub_duration":0.079233095000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":939,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.559831+0000","last_change":"2024-03-30T07:45:04.561495+0000","last_active":"2024-03-30T07:45:04.559831+0000","last_peered":"2024-03-30T07:45:04.559831+0000","last_clean":"2024-03-30T07:45:04.559831+0000","last_became_active":"2024-03-30T07:44:18.148922+0000","last_became_peered":"2024-03-30T07:44:18.148922+0000","last_unstale":"2024-03-30T07:45:04.559831+0000","last_undegraded":"2024-03-30T07:45:04.559831+0000","last_fullsized":"2024-03-30T07:45:04.559831+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.559009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.559009+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.559009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:16:59.538322+0000","scrub_duration":0.065331189999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":925,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:20.852182+0000","last_change":"2024-03-30T07:45:20.853916+0000","last_active":"2024-03-30T07:45:20.852182+0000","last_peered":"2024-03-30T07:45:20.852182+0000","last_clean":"2024-03-30T07:45:20.852182+0000","last_became_active":"2024-03-30T07:18:04.154117+0000","last_became_peered":"2024-03-30T07:18:04.154117+0000","last_unstale":"2024-03-30T07:45:20.852182+0000","last_undegraded":"2024-03-30T07:45:20.852182+0000","last_fullsized":"2024-03-30T07:45:20.852182+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:20.851358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:20.851358+0000","last_clean_scrub_stamp":"2024-03-30T07:45:20.851358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:29:25.700656+0000","scrub_duration":0.062714860999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":948,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:01.661133+0000","last_change":"2024-03-30T07:45:01.667042+0000","last_active":"2024-03-30T07:45:01.661133+0000","last_peered":"2024-03-30T07:45:01.661133+0000","last_clean":"2024-03-30T07:45:01.661133+0000","last_became_active":"2024-03-30T07:24:14.242943+0000","last_became_peered":"2024-03-30T07:24:14.242943+0000","last_unstale":"2024-03-30T07:45:01.661133+0000","last_undegraded":"2024-03-30T07:45:01.661133+0000","last_fullsized":"2024-03-30T07:45:01.661133+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:01.660355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:01.660355+0000","last_clean_scrub_stamp":"2024-03-30T07:45:01.660355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:49:18.855837+0000","scrub_duration":0.041866140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":945,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:06.144826+0000","last_change":"2024-03-30T07:45:06.150467+0000","last_active":"2024-03-30T07:45:06.144826+0000","last_peered":"2024-03-30T07:45:06.144826+0000","last_clean":"2024-03-30T07:45:06.144826+0000","last_became_active":"2024-03-30T07:18:53.668328+0000","last_became_peered":"2024-03-30T07:18:53.668328+0000","last_unstale":"2024-03-30T07:45:06.144826+0000","last_undegraded":"2024-03-30T07:45:06.144826+0000","last_fullsized":"2024-03-30T07:45:06.144826+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:06.144033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:06.144033+0000","last_clean_scrub_stamp":"2024-03-30T07:45:06.144033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:39.885228+0000","scrub_duration":0.083749650999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":964,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.953189+0000","last_change":"2024-03-30T07:45:02.955292+0000","last_active":"2024-03-30T07:45:02.953189+0000","last_peered":"2024-03-30T07:45:02.953189+0000","last_clean":"2024-03-30T07:45:02.953189+0000","last_became_active":"2024-03-30T07:18:37.324747+0000","last_became_peered":"2024-03-30T07:18:37.324747+0000","last_unstale":"2024-03-30T07:45:02.953189+0000","last_undegraded":"2024-03-30T07:45:02.953189+0000","last_fullsized":"2024-03-30T07:45:02.953189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.952382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.952382+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.952382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:26:26.189127+0000","scrub_duration":0.037306020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":921,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:10.906081+0000","last_change":"2024-03-30T07:45:10.907928+0000","last_active":"2024-03-30T07:45:10.906081+0000","last_peered":"2024-03-30T07:45:10.906081+0000","last_clean":"2024-03-30T07:45:10.906081+0000","last_became_active":"2024-03-30T07:18:16.739071+0000","last_became_peered":"2024-03-30T07:18:16.739071+0000","last_unstale":"2024-03-30T07:45:10.906081+0000","last_undegraded":"2024-03-30T07:45:10.906081+0000","last_fullsized":"2024-03-30T07:45:10.906081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:10.905264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:10.905264+0000","last_clean_scrub_stamp":"2024-03-30T07:45:10.905264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:58:09.541385+0000","scrub_duration":0.054738342000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":922,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:08.853422+0000","last_change":"2024-03-30T07:45:08.856035+0000","last_active":"2024-03-30T07:45:08.853422+0000","last_peered":"2024-03-30T07:45:08.853422+0000","last_clean":"2024-03-30T07:45:08.853422+0000","last_became_active":"2024-03-30T07:18:16.036419+0000","last_became_peered":"2024-03-30T07:18:16.036419+0000","last_unstale":"2024-03-30T07:45:08.853422+0000","last_undegraded":"2024-03-30T07:45:08.853422+0000","last_fullsized":"2024-03-30T07:45:08.853422+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:08.852576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:08.852576+0000","last_clean_scrub_stamp":"2024-03-30T07:45:08.852576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T16:57:33.949616+0000","scrub_duration":0.032762352000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":927,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:21.893438+0000","last_change":"2024-03-30T07:45:21.893438+0000","last_active":"2024-03-30T07:45:21.893438+0000","last_peered":"2024-03-30T07:45:21.893438+0000","last_clean":"2024-03-30T07:45:21.893438+0000","last_became_active":"2024-03-30T07:18:03.157161+0000","last_became_peered":"2024-03-30T07:18:03.157161+0000","last_unstale":"2024-03-30T07:45:21.893438+0000","last_undegraded":"2024-03-30T07:45:21.893438+0000","last_fullsized":"2024-03-30T07:45:21.893438+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:21.886696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:21.886696+0000","last_clean_scrub_stamp":"2024-03-30T07:45:21.886696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:08:16.332617+0000","scrub_duration":0.061900677000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":959,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:02.079416+0000","last_change":"2024-03-30T07:45:02.085105+0000","last_active":"2024-03-30T07:45:02.079416+0000","last_peered":"2024-03-30T07:45:02.079416+0000","last_clean":"2024-03-30T07:45:02.079416+0000","last_became_active":"2024-03-30T07:44:18.298624+0000","last_became_peered":"2024-03-30T07:44:18.298624+0000","last_unstale":"2024-03-30T07:45:02.079416+0000","last_undegraded":"2024-03-30T07:45:02.079416+0000","last_fullsized":"2024-03-30T07:45:02.079416+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:02.078616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:02.078616+0000","last_clean_scrub_stamp":"2024-03-30T07:45:02.078616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:28:10.960802+0000","scrub_duration":0.063356277000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"994'8","reported_seq":955,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:00.624731+0000","last_change":"2024-03-30T07:45:00.630869+0000","last_active":"2024-03-30T07:45:00.624731+0000","last_peered":"2024-03-30T07:45:00.624731+0000","last_clean":"2024-03-30T07:45:00.624731+0000","last_became_active":"2024-03-30T07:44:18.254312+0000","last_became_peered":"2024-03-30T07:44:18.254312+0000","last_unstale":"2024-03-30T07:45:00.624731+0000","last_undegraded":"2024-03-30T07:45:00.624731+0000","last_fullsized":"2024-03-30T07:45:00.624731+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":0,"last_scrub":"994'8","last_scrub_stamp":"2024-03-30T07:45:00.623836+0000","last_deep_scrub":"994'8","last_deep_scrub_stamp":"2024-03-30T07:45:00.623836+0000","last_clean_scrub_stamp":"2024-03-30T07:45:00.623836+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:48:12.552105+0000","scrub_duration":0.150706335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":948,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-30T07:45:04.223038+0000","last_change":"2024-03-30T07:45:04.228572+0000","last_active":"2024-03-30T07:45:04.223038+0000","last_peered":"2024-03-30T07:45:04.223038+0000","last_clean":"2024-03-30T07:45:04.223038+0000","last_became_active":"2024-03-30T07:24:15.793263+0000","last_became_peered":"2024-03-30T07:24:15.793263+0000","last_unstale":"2024-03-30T07:45:04.223038+0000","last_undegraded":"2024-03-30T07:45:04.223038+0000","last_fullsized":"2024-03-30T07:45:04.223038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T07:45:04.222263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T07:45:04.222263+0000","last_clean_scrub_stamp":"2024-03-30T07:45:04.222263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:27:33.606081+0000","scrub_duration":0.066534888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":2,"up_from":22,"seq":94489280866,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64652,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94307188,"statfs":{"total":96636764160,"available":96570560512,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:45:25 2024","interfaces":[{"interface":"back","average":{"1min":13.898999999999999,"5min":16.204999999999998,"15min":19.192},"min":{"1min":4.7050000000000001,"5min":4.7050000000000001,"15min":4.7050000000000001},"max":{"1min":118.928,"5min":249.63200000000001,"15min":292.04199999999997},"last":11.468},{"interface":"front","average":{"1min":16.268999999999998,"5min":19.343,"15min":20.385000000000002},"min":{"1min":6.6539999999999999,"5min":5.3339999999999996,"15min":4.79},"max":{"1min":99.733000000000004,"5min":248.767,"15min":299.35000000000002},"last":10.032}]},{"osd":1,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":11.307,"5min":13.699,"15min":17.414000000000001},"min":{"1min":6.0860000000000003,"5min":5.0430000000000001,"15min":5.0430000000000001},"max":{"1min":24.446999999999999,"5min":104.782,"15min":169.17099999999999},"last":7.1159999999999997},{"interface":"front","average":{"1min":16.443999999999999,"5min":19.584,"15min":24.741},"min":{"1min":7.851,"5min":7.851,"15min":6.7789999999999999},"max":{"1min":60.030999999999999,"5min":181.46199999999999,"15min":286.52199999999999},"last":11.19}]},{"osd":3,"last update":"Sat Mar 30 07:45:23 2024","interfaces":[{"interface":"back","average":{"1min":1478.0139999999999,"5min":818.93700000000001,"15min":325.38},"min":{"1min":7.2729999999999997,"5min":7.2729999999999997,"15min":7.2729999999999997},"max":{"1min":13517.566000000001,"5min":14020.602000000001,"15min":14020.602000000001},"last":10.603999999999999},{"interface":"front","average":{"1min":1423.404,"5min":814.928,"15min":326.577},"min":{"1min":13.592000000000001,"5min":11.776999999999999,"15min":10.022},"max":{"1min":13210.786,"5min":13714.877,"15min":13714.877},"last":15.265000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:45:02 2024","interfaces":[{"interface":"back","average":{"1min":1421.798,"5min":613.84400000000005,"15min":308.387},"min":{"1min":7.923,"5min":7.4749999999999996,"15min":6.5979999999999999},"max":{"1min":12463.806,"5min":12556.546,"15min":12556.546},"last":12.081},{"interface":"front","average":{"1min":1426.5699999999999,"5min":619.529,"15min":314.673},"min":{"1min":12.487,"5min":8.8249999999999993,"15min":8.3300000000000001},"max":{"1min":12465.362999999999,"5min":12557.409,"15min":12557.409},"last":14.016}]},{"osd":5,"last update":"Sat Mar 30 07:44:56 2024","interfaces":[{"interface":"back","average":{"1min":27.942,"5min":27.231999999999999,"15min":51.024000000000001},"min":{"1min":9.3350000000000009,"5min":9.3350000000000009,"15min":9.3350000000000009},"max":{"1min":161.35400000000001,"5min":254.04300000000001,"15min":4768.4960000000001},"last":14.691000000000001},{"interface":"front","average":{"1min":32.487000000000002,"5min":32.831000000000003,"15min":51.594999999999999},"min":{"1min":12.095000000000001,"5min":12.095000000000001,"15min":7.8760000000000003},"max":{"1min":165.96600000000001,"5min":321.42099999999999,"15min":4538.7879999999996},"last":17.798999999999999}]},{"osd":6,"last update":"Sat Mar 30 07:45:19 2024","interfaces":[{"interface":"back","average":{"1min":16.044,"5min":16.044,"15min":16.044},"min":{"1min":7.2359999999999998,"5min":7.2359999999999998,"15min":7.2359999999999998},"max":{"1min":100.77,"5min":100.77,"15min":100.77},"last":12.762},{"interface":"front","average":{"1min":20.059999999999999,"5min":20.059999999999999,"15min":20.059999999999999},"min":{"1min":15.148,"5min":15.148,"15min":15.148},"max":{"1min":42.055999999999997,"5min":42.055999999999997,"15min":42.055999999999997},"last":18.638999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:45:08 2024","interfaces":[{"interface":"back","average":{"1min":22.399000000000001,"5min":21.151,"15min":337.69299999999998},"min":{"1min":7.9219999999999997,"5min":6.7649999999999997,"15min":6.7649999999999997},"max":{"1min":179.583,"5min":334.75400000000002,"15min":19293.749},"last":9.4239999999999995},{"interface":"front","average":{"1min":27.184999999999999,"5min":23.873999999999999,"15min":340.09699999999998},"min":{"1min":10.199,"5min":9.1430000000000007,"15min":8.7059999999999995},"max":{"1min":194.93299999999999,"5min":205.19800000000001,"15min":19292.978999999999},"last":13.439}]}]},{"osd":0,"up_from":22,"seq":94489280867,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":41.609999999999999,"5min":27.103000000000002,"15min":22.817},"min":{"1min":5.0839999999999996,"5min":4.5830000000000002,"15min":4.5830000000000002},"max":{"1min":340.18299999999999,"5min":340.18299999999999,"15min":340.18299999999999},"last":8.8019999999999996},{"interface":"front","average":{"1min":47.844999999999999,"5min":25.233000000000001,"15min":20.251000000000001},"min":{"1min":5.9630000000000001,"5min":5.5140000000000002,"15min":5.2130000000000001},"max":{"1min":339.76100000000002,"5min":339.76100000000002,"15min":339.76100000000002},"last":10.090999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":34.978000000000002,"5min":26.463999999999999,"15min":25.451000000000001},"min":{"1min":7.351,"5min":7.351,"15min":6.6619999999999999},"max":{"1min":343.45699999999999,"5min":443.25700000000001,"15min":443.25700000000001},"last":9.4570000000000007},{"interface":"front","average":{"1min":39.024000000000001,"5min":29.751000000000001,"15min":32.283000000000001},"min":{"1min":9.9499999999999993,"5min":8.8550000000000004,"15min":8.8550000000000004},"max":{"1min":346.71300000000002,"5min":451.50599999999997,"15min":451.50599999999997},"last":10.878}]},{"osd":3,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":1499.876,"5min":472.52499999999998,"15min":195.184},"min":{"1min":9.7899999999999991,"5min":9.7899999999999991,"15min":6.6429999999999998},"max":{"1min":13546.239,"5min":13546.239,"15min":13546.239},"last":12.177},{"interface":"front","average":{"1min":1557.163,"5min":474.86099999999999,"15min":194.38},"min":{"1min":7.4000000000000004,"5min":6.7530000000000001,"15min":5.2569999999999997},"max":{"1min":13851.101000000001,"5min":13851.101000000001,"15min":13851.101000000001},"last":12.471}]},{"osd":4,"last update":"Sat Mar 30 07:44:45 2024","interfaces":[{"interface":"back","average":{"1min":1914.47,"5min":802.899,"15min":286.38499999999999},"min":{"1min":10.516,"5min":10.186,"15min":5.8869999999999996},"max":{"1min":13561.277,"5min":13561.277,"15min":13561.277},"last":11.5},{"interface":"front","average":{"1min":1913.037,"5min":809.60299999999995,"15min":286.928},"min":{"1min":6.1310000000000002,"5min":6.1310000000000002,"15min":6.1310000000000002},"max":{"1min":13560.536,"5min":13560.536,"15min":13560.536},"last":14.458}]},{"osd":5,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":41.381999999999998,"5min":25.181000000000001,"15min":60.106000000000002},"min":{"1min":7.4539999999999997,"5min":6.8600000000000003,"15min":6.8600000000000003},"max":{"1min":460.38799999999998,"5min":460.38799999999998,"15min":7478.7280000000001},"last":13.222},{"interface":"front","average":{"1min":47.198,"5min":34.844999999999999,"15min":61.795000000000002},"min":{"1min":10.906000000000001,"5min":10.906000000000001,"15min":7.125},"max":{"1min":470.99599999999998,"5min":470.99599999999998,"15min":7477.2579999999998},"last":18.922000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:45:20 2024","interfaces":[{"interface":"back","average":{"1min":20.995000000000001,"5min":20.995000000000001,"15min":20.995000000000001},"min":{"1min":12.752000000000001,"5min":12.752000000000001,"15min":12.752000000000001},"max":{"1min":38.423000000000002,"5min":38.423000000000002,"15min":38.423000000000002},"last":19.556999999999999},{"interface":"front","average":{"1min":16.931000000000001,"5min":16.931000000000001,"15min":16.931000000000001},"min":{"1min":9.7100000000000009,"5min":9.7100000000000009,"15min":9.7100000000000009},"max":{"1min":37.646999999999998,"5min":37.646999999999998,"15min":37.646999999999998},"last":20.247}]},{"osd":7,"last update":"Sat Mar 30 07:45:07 2024","interfaces":[{"interface":"back","average":{"1min":15.141999999999999,"5min":20.492999999999999,"15min":327.12400000000002},"min":{"1min":7.2869999999999999,"5min":7.1479999999999997,"15min":5.8789999999999996},"max":{"1min":38.997,"5min":295.38600000000002,"15min":21868.147000000001},"last":16.940999999999999},{"interface":"front","average":{"1min":21.547999999999998,"5min":23.123999999999999,"15min":330.87700000000001},"min":{"1min":11.052,"5min":10.34,"15min":9.3140000000000001},"max":{"1min":66.225999999999999,"5min":302.11500000000001,"15min":21842.699000000001},"last":13.829000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346261,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78604,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94293236,"statfs":{"total":96636764160,"available":96556273664,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:50 2024","interfaces":[{"interface":"back","average":{"1min":42.113,"5min":30.841000000000001,"15min":21.015000000000001},"min":{"1min":5.0019999999999998,"5min":4.7960000000000003,"15min":4.7960000000000003},"max":{"1min":193.56299999999999,"5min":350.36599999999999,"15min":350.36599999999999},"last":7.6200000000000001},{"interface":"front","average":{"1min":39.423000000000002,"5min":29.962,"15min":26.541},"min":{"1min":6.9080000000000004,"5min":5.9379999999999997,"15min":5.9379999999999997},"max":{"1min":196.321,"5min":410.93099999999998,"15min":410.93099999999998},"last":8.9640000000000004}]},{"osd":1,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":26.478999999999999,"5min":24.071999999999999,"15min":20.161999999999999},"min":{"1min":5.5709999999999997,"5min":4.6840000000000002,"15min":4.6840000000000002},"max":{"1min":137.941,"5min":313.54399999999998,"15min":313.54399999999998},"last":10.678000000000001},{"interface":"front","average":{"1min":33.582000000000001,"5min":33.255000000000003,"15min":26.905999999999999},"min":{"1min":8.0679999999999996,"5min":8.0679999999999996,"15min":6.3719999999999999},"max":{"1min":138.87200000000001,"5min":466.54599999999999,"15min":466.54599999999999},"last":15.638999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":36.649000000000001,"5min":24.920999999999999,"15min":25.821999999999999},"min":{"1min":7.6799999999999997,"5min":6.1139999999999999,"15min":6.1139999999999999},"max":{"1min":198.03100000000001,"5min":371.74200000000002,"15min":768.79100000000005},"last":11.449},{"interface":"front","average":{"1min":41.881,"5min":29.448,"15min":25.213000000000001},"min":{"1min":9.2889999999999997,"5min":8.6340000000000003,"15min":6.2720000000000002},"max":{"1min":200.797,"5min":372.322,"15min":372.322},"last":12.689}]},{"osd":3,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":3380.04,"5min":1086.2429999999999,"15min":410.63499999999999},"min":{"1min":9.2530000000000001,"5min":8.2789999999999999,"15min":6.3659999999999997},"max":{"1min":13684.860000000001,"5min":13684.860000000001,"15min":13684.860000000001},"last":12.090999999999999},{"interface":"front","average":{"1min":3346.288,"5min":1072.2090000000001,"15min":407.68000000000001},"min":{"1min":10.25,"5min":7.7939999999999996,"15min":7.6120000000000001},"max":{"1min":13604.335999999999,"5min":13604.335999999999,"15min":13604.335999999999},"last":16.292000000000002}]},{"osd":5,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":34.985999999999997,"5min":31.134,"15min":43.316000000000003},"min":{"1min":11.901999999999999,"5min":7.9660000000000002,"15min":6.9180000000000001},"max":{"1min":202.54400000000001,"5min":421.40600000000001,"15min":4807.1080000000002},"last":13.321999999999999},{"interface":"front","average":{"1min":32.462000000000003,"5min":31.606000000000002,"15min":46.908000000000001},"min":{"1min":9.0909999999999993,"5min":7.5350000000000001,"15min":7.0830000000000002},"max":{"1min":140.81399999999999,"5min":528.52099999999996,"15min":4805.7799999999997},"last":17.547000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":40.976999999999997,"5min":40.976999999999997,"15min":40.976999999999997},"min":{"1min":10.939,"5min":10.939,"15min":10.939},"max":{"1min":201.977,"5min":201.977,"15min":201.977},"last":15.025},{"interface":"front","average":{"1min":37.075000000000003,"5min":37.075000000000003,"15min":37.075000000000003},"min":{"1min":10.868,"5min":10.868,"15min":10.868},"max":{"1min":144.673,"5min":144.673,"15min":144.673},"last":16.928000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":41.027999999999999,"5min":36.396999999999998,"15min":341.108},"min":{"1min":11.250999999999999,"5min":10.010999999999999,"15min":7.3540000000000001},"max":{"1min":219.53100000000001,"5min":530.32500000000005,"15min":20845.164000000001},"last":19.782},{"interface":"front","average":{"1min":39.600999999999999,"5min":46.209000000000003,"15min":345.48099999999999},"min":{"1min":9.8770000000000007,"5min":8.9100000000000001,"15min":7.5060000000000002},"max":{"1min":190.62799999999999,"5min":664.02300000000002,"15min":20888.814999999999},"last":19.158999999999999}]}]},{"osd":7,"up_from":19,"seq":81604378961,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60556,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311284,"statfs":{"total":96636764160,"available":96574754816,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":10.182,"5min":18.259,"15min":21.504000000000001},"min":{"1min":4.8689999999999998,"5min":4.8689999999999998,"15min":4.2789999999999999},"max":{"1min":14.627000000000001,"5min":358.94499999999999,"15min":368.16899999999998},"last":14.698},{"interface":"front","average":{"1min":9.3840000000000003,"5min":17.870999999999999,"15min":24.042999999999999},"min":{"1min":5.7149999999999999,"5min":4.7430000000000003,"15min":4.7430000000000003},"max":{"1min":23.771999999999998,"5min":343.65699999999998,"15min":349.97199999999998},"last":10.419}]},{"osd":1,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":9.157,"5min":12.548999999999999,"15min":23.257000000000001},"min":{"1min":4.8639999999999999,"5min":4.577,"15min":4.577},"max":{"1min":18.594000000000001,"5min":84.540000000000006,"15min":472.33300000000003},"last":11.228},{"interface":"front","average":{"1min":13.550000000000001,"5min":18.526,"15min":25.012},"min":{"1min":9.4909999999999997,"5min":7.4370000000000003,"15min":6.2320000000000002},"max":{"1min":21.835999999999999,"5min":184.33500000000001,"15min":292.92099999999999},"last":16.350000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:45:04 2024","interfaces":[{"interface":"back","average":{"1min":13.602,"5min":23.742000000000001,"15min":31.006},"min":{"1min":8.6899999999999995,"5min":8.4410000000000007,"15min":7.1180000000000003},"max":{"1min":18.978999999999999,"5min":341.96100000000001,"15min":400.43099999999998},"last":17.126999999999999},{"interface":"front","average":{"1min":20.641999999999999,"5min":30.042999999999999,"15min":34.962000000000003},"min":{"1min":12.734,"5min":10.554,"15min":8.0380000000000003},"max":{"1min":103.166,"5min":389.37200000000001,"15min":439.31999999999999},"last":17.890999999999998}]},{"osd":3,"last update":"Sat Mar 30 07:45:27 2024","interfaces":[{"interface":"back","average":{"1min":744.43799999999999,"5min":594.50900000000001,"15min":252.25200000000001},"min":{"1min":11.084,"5min":11.084,"15min":7.5940000000000003},"max":{"1min":8536.5470000000005,"5min":13240.941000000001,"15min":13240.941000000001},"last":18.657},{"interface":"front","average":{"1min":798.27599999999995,"5min":607.12,"15min":256.78100000000001},"min":{"1min":7.2190000000000003,"5min":7.2190000000000003,"15min":7.2190000000000003},"max":{"1min":8892.6100000000006,"5min":13596.368,"15min":13596.368},"last":12.486000000000001}]},{"osd":4,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":1875.7170000000001,"5min":716.80100000000004,"15min":291.15100000000001},"min":{"1min":11.651999999999999,"5min":6.4130000000000003,"15min":5.5620000000000003},"max":{"1min":14214.021000000001,"5min":14214.021000000001,"15min":14214.021000000001},"last":10.993},{"interface":"front","average":{"1min":1866.454,"5min":725.48099999999999,"15min":295.13099999999997},"min":{"1min":6.1879999999999997,"5min":5.5119999999999996,"15min":5.5119999999999996},"max":{"1min":14206.152,"5min":14206.152,"15min":14206.152},"last":19.411000000000001}]},{"osd":5,"last update":"Sat Mar 30 07:45:14 2024","interfaces":[{"interface":"back","average":{"1min":22.273,"5min":25.465,"15min":108.16500000000001},"min":{"1min":12.196999999999999,"5min":8.0340000000000007,"15min":7.4050000000000002},"max":{"1min":105.785,"5min":345.90499999999997,"15min":7694.1369999999997},"last":21.055},{"interface":"front","average":{"1min":19.018000000000001,"5min":20.571000000000002,"15min":112.063},"min":{"1min":8.3079999999999998,"5min":6.7869999999999999,"15min":6.7869999999999999},"max":{"1min":104.148,"5min":203.31299999999999,"15min":7725.4070000000002},"last":15.496}]},{"osd":6,"last update":"Sat Mar 30 07:45:19 2024","interfaces":[{"interface":"back","average":{"1min":22.693999999999999,"5min":22.693999999999999,"15min":22.693999999999999},"min":{"1min":13.304,"5min":13.304,"15min":13.304},"max":{"1min":106.682,"5min":106.682,"15min":106.682},"last":20.276},{"interface":"front","average":{"1min":15.510999999999999,"5min":15.510999999999999,"15min":15.510999999999999},"min":{"1min":9.7569999999999997,"5min":9.7569999999999997,"15min":9.7569999999999997},"max":{"1min":27.407,"5min":27.407,"15min":27.407},"last":13.837999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477087,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60952,"kb_used_data":7696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310888,"statfs":{"total":96636764160,"available":96574349312,"internally_reserved":0,"allocated":7880704,"data_stored":4551427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":13.853999999999999,"5min":12.869,"15min":18.533999999999999},"min":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":4.8620000000000001},"max":{"1min":55.762,"5min":132.02699999999999,"15min":421.78699999999998},"last":7.3280000000000003},{"interface":"front","average":{"1min":12.318,"5min":12.644,"15min":18.782},"min":{"1min":7.5750000000000002,"5min":5.2759999999999998,"15min":5.2009999999999996},"max":{"1min":40.883000000000003,"5min":78.022999999999996,"15min":409.91000000000003},"last":7.9470000000000001}]},{"osd":2,"last update":"Sat Mar 30 07:45:16 2024","interfaces":[{"interface":"back","average":{"1min":10.548,"5min":13.284000000000001,"15min":21.007000000000001},"min":{"1min":5.8360000000000003,"5min":4.9939999999999998,"15min":4.9939999999999998},"max":{"1min":30.565000000000001,"5min":180.42599999999999,"15min":433.34899999999999},"last":12.494999999999999},{"interface":"front","average":{"1min":14.967000000000001,"5min":17.143000000000001,"15min":23.596},"min":{"1min":8.5860000000000003,"5min":8.1790000000000003,"15min":5.6619999999999999},"max":{"1min":26.742999999999999,"5min":177.982,"15min":624.27300000000002},"last":13.661}]},{"osd":3,"last update":"Sat Mar 30 07:45:35 2024","interfaces":[{"interface":"back","average":{"1min":14.75,"5min":557.173,"15min":224.328},"min":{"1min":8.9450000000000003,"5min":7.4219999999999997,"15min":7.4219999999999997},"max":{"1min":31.649000000000001,"5min":12913.004000000001,"15min":12913.004000000001},"last":13.064},{"interface":"front","average":{"1min":12.099,"5min":565.59299999999996,"15min":225.97900000000001},"min":{"1min":7.9729999999999999,"5min":7.8120000000000003,"15min":6.7080000000000002},"max":{"1min":21.16,"5min":13136.255999999999,"15min":13136.255999999999},"last":9.2159999999999993}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":945.70399999999995,"5min":509.334,"15min":250.96799999999999},"min":{"1min":10.207000000000001,"5min":9.5410000000000004,"15min":9.5410000000000004},"max":{"1min":13045.378000000001,"5min":13045.378000000001,"15min":13045.378000000001},"last":14.254},{"interface":"front","average":{"1min":945.62400000000002,"5min":505.50599999999997,"15min":248.53200000000001},"min":{"1min":11.148,"5min":8.9190000000000005,"15min":6.9260000000000002},"max":{"1min":13043.406000000001,"5min":13043.406000000001,"15min":13043.406000000001},"last":9.7669999999999995}]},{"osd":5,"last update":"Sat Mar 30 07:44:53 2024","interfaces":[{"interface":"back","average":{"1min":16.731000000000002,"5min":19.794,"15min":46.238},"min":{"1min":9.8879999999999999,"5min":9.8879999999999999,"15min":8.3439999999999994},"max":{"1min":32.831000000000003,"5min":192.34200000000001,"15min":7340.1670000000004},"last":14.977},{"interface":"front","average":{"1min":15.535,"5min":17.545999999999999,"15min":44.427999999999997},"min":{"1min":10.175000000000001,"5min":10.173999999999999,"15min":6.6260000000000003},"max":{"1min":22.318999999999999,"5min":123.297,"15min":7345.2740000000003},"last":10.859}]},{"osd":6,"last update":"Sat Mar 30 07:45:17 2024","interfaces":[{"interface":"back","average":{"1min":21.728000000000002,"5min":21.728000000000002,"15min":21.728000000000002},"min":{"1min":11.571,"5min":11.571,"15min":11.571},"max":{"1min":73.358999999999995,"5min":73.358999999999995,"15min":73.358999999999995},"last":11.429},{"interface":"front","average":{"1min":24.260000000000002,"5min":24.260000000000002,"15min":24.260000000000002},"min":{"1min":7.6379999999999999,"5min":7.6379999999999999,"15min":7.6379999999999999},"max":{"1min":246.25200000000001,"5min":246.25200000000001,"15min":246.25200000000001},"last":16.497}]},{"osd":7,"last update":"Sat Mar 30 07:45:16 2024","interfaces":[{"interface":"back","average":{"1min":19.524999999999999,"5min":16.539000000000001,"15min":242.172},"min":{"1min":9.8439999999999994,"5min":8.2170000000000005,"15min":7.2149999999999999},"max":{"1min":60.286000000000001,"5min":115.461,"15min":20279.343000000001},"last":18.152999999999999},{"interface":"front","average":{"1min":16.806999999999999,"5min":21.263999999999999,"15min":250.071},"min":{"1min":9.2230000000000008,"5min":6.4550000000000001,"15min":6.4550000000000001},"max":{"1min":58.329999999999998,"5min":281.423,"15min":20311.815999999999},"last":17.088000000000001}]}]},{"osd":5,"up_from":231,"seq":992137445707,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60948,"kb_used_data":7692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94310892,"statfs":{"total":96636764160,"available":96574353408,"internally_reserved":0,"allocated":7876608,"data_stored":4551422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.442,"5min":15.161,"15min":17.379999999999999},"min":{"1min":4.9800000000000004,"5min":4.9160000000000004,"15min":4.7530000000000001},"max":{"1min":82.299000000000007,"5min":105.372,"15min":259.14499999999998},"last":12.090999999999999},{"interface":"front","average":{"1min":12.835000000000001,"5min":12.6,"15min":16.614000000000001},"min":{"1min":4.7300000000000004,"5min":4.7300000000000004,"15min":4.7300000000000004},"max":{"1min":87.344999999999999,"5min":115.901,"15min":351.22500000000002},"last":13.241}]},{"osd":1,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":14.042,"5min":26.402000000000001,"15min":23.454999999999998},"min":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"max":{"1min":45.496000000000002,"5min":988.86099999999999,"15min":988.86099999999999},"last":8.2899999999999991},{"interface":"front","average":{"1min":17.375,"5min":30.155999999999999,"15min":26.594999999999999},"min":{"1min":9.2240000000000002,"5min":8.4969999999999999,"15min":5.8979999999999997},"max":{"1min":83.228999999999999,"5min":910.32500000000005,"15min":910.32500000000005},"last":9.2379999999999995}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":16.353999999999999,"5min":17.457999999999998,"15min":21.376000000000001},"min":{"1min":9.2680000000000007,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":68.896000000000001,"5min":203.65000000000001,"15min":207.98400000000001},"last":9.8680000000000003},{"interface":"front","average":{"1min":19.318999999999999,"5min":19.245000000000001,"15min":21.574000000000002},"min":{"1min":8.6500000000000004,"5min":8.0069999999999997,"15min":6.8330000000000002},"max":{"1min":121.712,"5min":162.35900000000001,"15min":298.86900000000003},"last":11.481999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":2175.951,"5min":730.625,"15min":333.05200000000002},"min":{"1min":7.2910000000000004,"5min":6.7949999999999999,"15min":6.4710000000000001},"max":{"1min":12225.981,"5min":12600.021000000001,"15min":12600.021000000001},"last":13.824},{"interface":"front","average":{"1min":2150.0410000000002,"5min":728.16399999999999,"15min":331.73899999999998},"min":{"1min":11.061999999999999,"5min":8.5009999999999994,"15min":7.2210000000000001},"max":{"1min":12130.326999999999,"5min":12614.886,"15min":12614.886},"last":15.087}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":966.27599999999995,"5min":574.19799999999998,"15min":277.06999999999999},"min":{"1min":9.5980000000000008,"5min":7.4640000000000004,"15min":7.4640000000000004},"max":{"1min":12327.154,"5min":12774.885,"15min":12774.885},"last":17.452999999999999},{"interface":"front","average":{"1min":970.24699999999996,"5min":579.12599999999998,"15min":283.90300000000002},"min":{"1min":11.728,"5min":9.2469999999999999,"15min":9.2469999999999999},"max":{"1min":12333.322,"5min":12784.931,"15min":12784.931},"last":20.027000000000001}]},{"osd":6,"last update":"Sat Mar 30 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":16.585999999999999,"5min":16.585999999999999,"15min":16.585999999999999},"min":{"1min":6.9390000000000001,"5min":6.9390000000000001,"15min":6.9390000000000001},"max":{"1min":96.572999999999993,"5min":96.572999999999993,"15min":96.572999999999993},"last":16.317},{"interface":"front","average":{"1min":19.701000000000001,"5min":19.701000000000001,"15min":19.701000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":8.8109999999999999},"max":{"1min":97.343999999999994,"5min":97.343999999999994,"15min":97.343999999999994},"last":11.095000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:45:18 2024","interfaces":[{"interface":"back","average":{"1min":17.626999999999999,"5min":14.768000000000001,"15min":283.55200000000002},"min":{"1min":9.266,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":88.396000000000001,"5min":115.589,"15min":19723.973000000002},"last":18.408000000000001},{"interface":"front","average":{"1min":19.236000000000001,"5min":18.826000000000001,"15min":283.75799999999998},"min":{"1min":10.714,"5min":10.228999999999999,"15min":8.4640000000000004},"max":{"1min":74.75,"5min":84.802000000000007,"15min":19724.860000000001},"last":19.422000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542487,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82700,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94289140,"statfs":{"total":96636764160,"available":96552079360,"internally_reserved":0,"allocated":7475200,"data_stored":4153582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77199816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":48.765999999999998,"5min":27.518000000000001,"15min":21.376999999999999},"min":{"1min":5.6609999999999996,"5min":4.8179999999999996,"15min":4.8179999999999996},"max":{"1min":565.17899999999997,"5min":565.17899999999997,"15min":565.17899999999997},"last":10.603},{"interface":"front","average":{"1min":31.677,"5min":23.501999999999999,"15min":21.385000000000002},"min":{"1min":5.2270000000000003,"5min":5.2270000000000003,"15min":5.2270000000000003},"max":{"1min":313.54199999999997,"5min":313.54199999999997,"15min":313.54199999999997},"last":12.632999999999999}]},{"osd":1,"last update":"Sat Mar 30 07:45:30 2024","interfaces":[{"interface":"back","average":{"1min":25.103999999999999,"5min":24.623999999999999,"15min":21.890000000000001},"min":{"1min":7.391,"5min":7.242,"15min":6.3899999999999997},"max":{"1min":227.48699999999999,"5min":291.73599999999999,"15min":291.73599999999999},"last":13.243},{"interface":"front","average":{"1min":21.98,"5min":30.763999999999999,"15min":25.815999999999999},"min":{"1min":6.9530000000000003,"5min":6.9530000000000003,"15min":6.415},"max":{"1min":159.37200000000001,"5min":541.21199999999999,"15min":541.21199999999999},"last":7.4720000000000004}]},{"osd":2,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":41.368000000000002,"5min":23.292000000000002,"15min":20.510000000000002},"min":{"1min":7.0460000000000003,"5min":6.5609999999999999,"15min":6.1269999999999998},"max":{"1min":510.08100000000002,"5min":510.08100000000002,"15min":510.08100000000002},"last":8.7789999999999999},{"interface":"front","average":{"1min":44.139000000000003,"5min":32.133000000000003,"15min":27.062000000000001},"min":{"1min":6.7709999999999999,"5min":6.7709999999999999,"15min":6.7709999999999999},"max":{"1min":549.73000000000002,"5min":549.73000000000002,"15min":549.73000000000002},"last":11.653}]},{"osd":4,"last update":"Sat Mar 30 07:44:41 2024","interfaces":[{"interface":"back","average":{"1min":1049.1210000000001,"5min":587.13699999999994,"15min":293.30000000000001},"min":{"1min":8.2110000000000003,"5min":7.7279999999999998,"15min":6.2309999999999999},"max":{"1min":12889.261,"5min":12889.261,"15min":12889.261},"last":9.3870000000000005},{"interface":"front","average":{"1min":1027.0509999999999,"5min":593.33100000000002,"15min":294.01900000000001},"min":{"1min":8.9149999999999991,"5min":7.5570000000000004,"15min":7.5570000000000004},"max":{"1min":12891.540999999999,"5min":12891.540999999999,"15min":12891.540999999999},"last":13.849}]},{"osd":5,"last update":"Sat Mar 30 07:44:42 2024","interfaces":[{"interface":"back","average":{"1min":50.399000000000001,"5min":26.745999999999999,"15min":95.335999999999999},"min":{"1min":8.048,"5min":6.6710000000000003,"15min":6.6710000000000003},"max":{"1min":529.14099999999996,"5min":529.14099999999996,"15min":7698.9949999999999},"last":15.074999999999999},{"interface":"front","average":{"1min":37.965000000000003,"5min":24.984999999999999,"15min":98.561999999999998},"min":{"1min":9.532,"5min":9.532,"15min":8.5830000000000002},"max":{"1min":307.86000000000001,"5min":307.86000000000001,"15min":7917.3400000000001},"last":14.461}]},{"osd":6,"last update":"Sat Mar 30 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":34.106000000000002,"5min":34.106000000000002,"15min":34.106000000000002},"min":{"1min":10.101000000000001,"5min":10.101000000000001,"15min":10.101000000000001},"max":{"1min":172.33199999999999,"5min":172.33199999999999,"15min":172.33199999999999},"last":17.41},{"interface":"front","average":{"1min":39.170999999999999,"5min":39.170999999999999,"15min":39.170999999999999},"min":{"1min":12.711,"5min":12.711,"15min":12.711},"max":{"1min":248.833,"5min":248.833,"15min":248.833},"last":15.726000000000001}]},{"osd":7,"last update":"Sat Mar 30 07:45:22 2024","interfaces":[{"interface":"back","average":{"1min":41.500999999999998,"5min":29.702000000000002,"15min":287},"min":{"1min":11.760999999999999,"5min":9.9580000000000002,"15min":9.5519999999999996},"max":{"1min":295.31599999999997,"5min":527.00300000000004,"15min":20495.457999999999},"last":19.128},{"interface":"front","average":{"1min":39.914999999999999,"5min":29.753,"15min":288.40300000000002},"min":{"1min":11.737,"5min":11.025,"15min":11.025},"max":{"1min":262.07799999999997,"5min":308.75900000000001,"15min":20496.241000000002},"last":18.033000000000001}]}]},{"osd":6,"up_from":978,"seq":4200478015821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60560,"kb_used_data":7304,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94311280,"statfs":{"total":96636764160,"available":96574750720,"internally_reserved":0,"allocated":7479296,"data_stored":4153587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":17.824000000000002,"5min":17.824000000000002,"15min":17.824000000000002},"min":{"1min":6.6159999999999997,"5min":6.6159999999999997,"15min":6.6159999999999997},"max":{"1min":125.791,"5min":125.791,"15min":125.791},"last":9.5020000000000007},{"interface":"front","average":{"1min":26.898,"5min":26.898,"15min":26.898},"min":{"1min":7.5309999999999997,"5min":7.5309999999999997,"15min":7.5309999999999997},"max":{"1min":265.62900000000002,"5min":265.62900000000002,"15min":265.62900000000002},"last":7.8529999999999998}]},{"osd":1,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":9.5139999999999993,"5min":9.5139999999999993,"15min":9.5139999999999993},"min":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"max":{"1min":20.934999999999999,"5min":20.934999999999999,"15min":20.934999999999999},"last":6.8289999999999997},{"interface":"front","average":{"1min":19.276,"5min":19.276,"15min":19.276},"min":{"1min":7.6959999999999997,"5min":7.6959999999999997,"15min":7.6959999999999997},"max":{"1min":128.578,"5min":128.578,"15min":128.578},"last":14.510999999999999}]},{"osd":2,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":19.367999999999999,"5min":19.367999999999999,"15min":19.367999999999999},"min":{"1min":8.2680000000000007,"5min":8.2680000000000007,"15min":8.2680000000000007},"max":{"1min":119.96299999999999,"5min":119.96299999999999,"15min":119.96299999999999},"last":13.731},{"interface":"front","average":{"1min":22.373999999999999,"5min":22.373999999999999,"15min":22.373999999999999},"min":{"1min":11.938000000000001,"5min":11.938000000000001,"15min":11.938000000000001},"max":{"1min":124.18300000000001,"5min":124.18300000000001,"15min":124.18300000000001},"last":15.420999999999999}]},{"osd":3,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":1112.355,"5min":1112.355,"15min":1112.355},"min":{"1min":6.9489999999999998,"5min":6.9489999999999998,"15min":6.9489999999999998},"max":{"1min":7852.915,"5min":7852.915,"15min":7852.915},"last":8.8200000000000003},{"interface":"front","average":{"1min":1060.037,"5min":1060.037,"15min":1060.037},"min":{"1min":8.3620000000000001,"5min":8.3620000000000001,"15min":8.3620000000000001},"max":{"1min":7655.6350000000002,"5min":7655.6350000000002,"15min":7655.6350000000002},"last":10.462}]},{"osd":4,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":2263.2840000000001,"5min":2263.2840000000001,"15min":2263.2840000000001},"min":{"1min":10.058999999999999,"5min":10.058999999999999,"15min":10.058999999999999},"max":{"1min":14228.966,"5min":14228.966,"15min":14228.966},"last":16.209},{"interface":"front","average":{"1min":2260.0839999999998,"5min":2260.0839999999998,"15min":2260.0839999999998},"min":{"1min":6.1109999999999998,"5min":6.1109999999999998,"15min":6.1109999999999998},"max":{"1min":14232.103999999999,"5min":14232.103999999999,"15min":14232.103999999999},"last":11.677}]},{"osd":5,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":21.741,"5min":21.741,"15min":21.741},"min":{"1min":11.446,"5min":11.446,"15min":11.446},"max":{"1min":125.30800000000001,"5min":125.30800000000001,"15min":125.30800000000001},"last":16.977},{"interface":"front","average":{"1min":17.327999999999999,"5min":17.327999999999999,"15min":17.327999999999999},"min":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"max":{"1min":105.018,"5min":105.018,"15min":105.018},"last":12.250999999999999}]},{"osd":7,"last update":"Sat Mar 30 07:45:15 2024","interfaces":[{"interface":"back","average":{"1min":31.873000000000001,"5min":31.873000000000001,"15min":31.873000000000001},"min":{"1min":11.654999999999999,"5min":11.654999999999999,"15min":11.654999999999999},"max":{"1min":246.78100000000001,"5min":246.78100000000001,"15min":246.78100000000001},"last":17.739999999999998},{"interface":"front","average":{"1min":22.234000000000002,"5min":22.234000000000002,"15min":22.234000000000002},"min":{"1min":10.59,"5min":10.59,"15min":10.59},"max":{"1min":107.79300000000001,"5min":107.79300000000001,"15min":107.79300000000001},"last":11.093999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T07:45:42.155 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.253+0000 7f49a456e640 10 monclient: get_monmap_and_config 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.253+0000 7f49a456e640 10 monclient: build_initial_monmap 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.253+0000 7f49a456e640 10 monclient: monmap: 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:45:42.254526+0000 2024-03-30T07:45:42.256 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:45:42.254526+0000 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a-legacy 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.52:6789/0 mon.noname-c-legacy 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-b 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:3: v2:172.21.15.47:3300/0 mon.noname-a 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:4: v2:172.21.15.52:3300/0 mon.noname-c 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.254+0000 7f49a456e640 1 Processor -- start 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.254+0000 7f49a456e640 1 -- start start 2024-03-30T07:45:42.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.254+0000 7f49a456e640 10 monclient: init 2024-03-30T07:45:42.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 0x7f499c0a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-b con 0x7f499c071be0 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f499c0b0ca0 addr v1:172.21.15.47:6789/0 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 1 --2- >> v2:172.21.15.52:3300/0 conn(0x7f499c13e820 0x7f499c13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-c con 0x7f499c13e820 addr v2:172.21.15.52:3300/0 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499c0a8cc0 con 0x7f499c0b0ca0 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 1 -- --> v2:172.21.15.52:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499c071fc0 con 0x7f499c13e820 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c0aa380 con 0x7f499c071be0 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a456e640 10 monclient(hunting): authenticate will time out at 3268.370284s 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a230a640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 0x7f499c0a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a1b09640 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f499c0b0ca0 0x7f499c144a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44930/0 (socket says 172.21.15.47:44930) 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a1b09640 1 -- 172.21.15.47:0/3545650016 learned_addr learned my addr 172.21.15.47:0/3545650016 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.255+0000 7f49a2b0b640 1 --2- 172.21.15.47:0/3545650016 >> v2:172.21.15.52:3300/0 conn(0x7f499c13e820 0x7f499c13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): get_auth_request con 0x7f499c071be0 auth_method 0 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): get_auth_request con 0x7f499c13e820 auth_method 0 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a230a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:42.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:42.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.256+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:42.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 10 monclient(hunting): handle_auth_done global_id 17153 payload 274 2024-03-30T07:45:42.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 1 -- 172.21.15.47:0/3545650016 >> v2:172.21.15.52:3300/0 conn(0x7f499c13e820 msgr2=0x7f499c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 1 --2- 172.21.15.47:0/3545650016 >> v2:172.21.15.52:3300/0 conn(0x7f499c13e820 0x7f499c13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 1 -- 172.21.15.47:0/3545650016 >> v1:172.21.15.47:6789/0 conn(0x7f499c0b0ca0 legacy=0x7f499c144a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 10 monclient: _finish_hunting 0 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 1 monclient: found mon.noname-b 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.258+0000 7f49a230a640 1 -- 172.21.15.47:0/3545650016 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c0a8970 con 0x7f499c071be0 2024-03-30T07:45:42.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.260+0000 7f49a230a640 1 --2- 172.21.15.47:0/3545650016 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 0x7f499c0a9150 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f4994009d90 tx=0x7f499402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36cea5101633a224 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 1 -- 172.21.15.47:0/3545650016 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499402cc50 con 0x7f499c071be0 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: dump: 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.267 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: _finish_auth 0 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: _check_auth_tickets 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:42.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 1 -- 172.21.15.47:0/3545650016 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499402cdb0 con 0x7f499c071be0 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 5 monclient: authenticate success, global_id 17153 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 10 monclient: get_monmap_and_config success 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 1 -- 172.21.15.47:0/3545650016 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994034660 con 0x7f499c071be0 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 4 set_mon_vals no callback set 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a1308640 10 monclient: dump: 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.269 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 10 monclient: shutdown 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 -- 172.21.15.47:0/3545650016 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 msgr2=0x7f499c0a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:42.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 --2- 172.21.15.47:0/3545650016 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 0x7f499c0a9150 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f4994009d90 tx=0x7f499402bea0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 -- 172.21.15.47:0/3545650016 shutdown_connections 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 --2- 172.21.15.47:0/3545650016 >> v2:172.21.15.52:3300/0 conn(0x7f499c13e820 0x7f499c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 --2- 172.21.15.47:0/3545650016 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c071be0 0x7f499c0a9150 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 -- 172.21.15.47:0/3545650016 >> 172.21.15.47:0/3545650016 conn(0x7f499c0b73d0 msgr2=0x7f499c0b77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 -- 172.21.15.47:0/3545650016 shutdown_connections 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.264+0000 7f49a456e640 1 -- 172.21.15.47:0/3545650016 wait complete. 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.265+0000 7f49a456e640 10 monclient: build_initial_monmap 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.265+0000 7f49a456e640 10 monclient: monmap: 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-03-30T07:45:42.271 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-03-30T07:45:42.272 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.272 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.noname-a 2024-03-30T07:45:42.272 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.noname-b 2024-03-30T07:45:42.272 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.noname-c 2024-03-30T07:45:42.272 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.265+0000 7f49a456e640 1 librados: starting msgr at 2024-03-30T07:45:42.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.265+0000 7f49a456e640 1 librados: starting objecter 2024-03-30T07:45:42.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 Processor -- start 2024-03-30T07:45:42.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 -- start start 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 librados: setting wanted keys 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 librados: calling monclient init 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient: init 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient: _reopen_session rank -1 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-a con 0x7f499c071be0 addr [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 --2- >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f499c0b0ca0 0x7f499c140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-b con 0x7f499c0b0ca0 addr [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): picked mon.noname-c con 0x7f499c13e820 addr [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c140380 con 0x7f499c071be0 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 -- --> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499c13f6b0 con 0x7f499c0b0ca0 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): start opening mon connection 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499c1406d0 con 0x7f499c13e820 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): _renew_subs 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.266+0000 7f49a456e640 10 monclient(hunting): authenticate will time out at 3268.381814s 2024-03-30T07:45:42.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a230a640 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:60312/0 (socket says 172.21.15.47:60312) 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 1 -- 172.21.15.47:0/3046948969 learned_addr learned my addr 172.21.15.47:0/3046948969 (peer_addr_for_me v2:172.21.15.47:0/0) 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 10 monclient(hunting): get_auth_request con 0x7f499c13e820 auth_method 0 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a2b0b640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a1b09640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f499c0b0ca0 0x7f499c140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a230a640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 msgr2=0x7f499c1d9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a230a640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 msgr2=0x7f499c1d9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a230a640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a230a640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a1b09640 10 monclient(hunting): get_auth_request con 0x7f499c0b0ca0 auth_method 0 2024-03-30T07:45:42.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a1b09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a1b09640 10 monclient(hunting): _init_auth method 2 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.267+0000 7f49a1b09640 10 monclient(hunting): _init_auth creating new auth 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a1b09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a1b09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.268+0000 7f49a1b09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 10 monclient(hunting): handle_auth_done global_id 17156 payload 824 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f499c0b0ca0 msgr2=0x7f499c140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f499c0b0ca0 0x7f499c140bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 msgr2=0x7f499c1d9a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 10 monclient: _finish_hunting 0 2024-03-30T07:45:42.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 monclient: found mon.noname-c 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.271+0000 7f49a2b0b640 1 -- 172.21.15.47:0/3046948969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994009a70 con 0x7f499c13e820 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.273+0000 7f49a2b0b640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e7f0 tx=0x7f498c00ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.273+0000 7f499affd640 10 client.?.objecter ms_handle_connect 0x7f499c13e820 2024-03-30T07:45:42.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.273+0000 7f499affd640 10 client.?.objecter resend_mon_ops 2024-03-30T07:45:42.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c009510 con 0x7f499c13e820 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: dump: 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: _finish_auth 0 2024-03-30T07:45:42.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: _check_auth_tickets 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00ee40 con 0x7f499c13e820 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: handle_config config(0 keys) v1 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00cb80 con 0x7f499c13e820 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 5 monclient: authenticate success, global_id 17156 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 monclient: _renew_subs 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c1416a0 con 0x7f499c13e820 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a0b07640 4 set_mon_vals no callback set 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f499affd640 10 monclient: dump: 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:fsid bc5c1745-fe46-4846-b9b2-29d88e4530d9 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-03-30T07:15:03.131639+0000 2024-03-30T07:45:42.284 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 18 (reef) 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:0: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.a 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:1: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.b 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 client.17156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 monclient: _renew_subs 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c136910 con 0x7f499c13e820 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 1 librados: init done 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.277+0000 7f49a456e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.278+0000 7f49a456e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.278+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499c140530 con 0x7f499c13e820 2024-03-30T07:45:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.538+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f498c00cce0 con 0x7f499c13e820 2024-03-30T07:45:42.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.538+0000 7f499affd640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 0x7f498806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T07:45:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.539+0000 7f49a230a640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 0x7f498806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T07:45:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.539+0000 7f49a230a640 10 monclient: get_auth_request con 0x7f498806c610 auth_method 0 2024-03-30T07:45:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.539+0000 7f499affd640 10 client.17156.objecter ms_handle_connect 0x7f498806c610 2024-03-30T07:45:42.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.539+0000 7f49a230a640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 0x7f498806ead0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4994036000 tx=0x7f4994030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(994..994 src has 601..994) v4 ==== 6294+0+0 (secure 0 0 0) 0x7f498c058900 con 0x7f499c13e820 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 10 client.17156.objecter ms_dispatch 0x7f499c000b90 osd_map(994..994 src has 601..994) v4 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 3 client.17156.objecter handle_osd_map got epochs [994,994] > 0 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 3 client.17156.objecter handle_osd_map decoding full epoch 994 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 20 client.17156.objecter dump_active .. 0 homeless 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181812 (secure 0 0 0) 0x7f498c054c40 con 0x7f499c13e820 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-30T07:45:42.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.551+0000 7f499affd640 10 monclient: _finish_command 1 = system:0 2024-03-30T07:45:42.613 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:10.424 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.617 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:10.427 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.631 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:10.442 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.668 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:10.445 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.669 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:10.446 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.679 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:10.456 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.686 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:10.496 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.690 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:10.501 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.701 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:10.512 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.758+0000 7f49a456e640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-30T07:45:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.758+0000 7f49a456e640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-30T07:45:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.758+0000 7f49a456e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.47:3301/0 2024-03-30T07:45:42.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:42.758+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f499c140530 con 0x7f499c13e820 2024-03-30T07:45:43.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f498c05c680 con 0x7f499c13e820 2024-03-30T07:45:43.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f499affd640 10 monclient: handle_config config(1 keys) v1 2024-03-30T07:45:43.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f49a0b07640 4 set_mon_vals no callback set 2024-03-30T07:45:43.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f49a0b07640 10 set_mon_vals mon_health_to_clog = false 2024-03-30T07:45:43.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f499affd640 1 -- 172.21.15.47:0/3046948969 <== mon.2 v2:172.21.15.47:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7f499c140530 con 0x7f499c13e820 2024-03-30T07:45:43.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f499affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-30T07:45:43.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.276+0000 7f499affd640 10 monclient: _finish_command 2 = system:0 2024-03-30T07:45:43.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 10 librados: watch_flush enter 2024-03-30T07:45:43.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 10 librados: watch_flush exit 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 20 client.17156.objecter shutdown clearing up homeless session... 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 10 client.17156.objecter successfully canceled tick 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 msgr2=0x7f498806ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 0x7f498806ead0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4994036000 tx=0x7f4994030000 comp rx=0 tx=0).stop 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 10 monclient: shutdown 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 msgr2=0x7f499c133120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e7f0 tx=0x7f498c00ebf0 comp rx=0 tx=0).stop 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 shutdown_connections 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:6800/34694,v1:172.21.15.52:6801/34694] conn(0x7f498806c610 0x7f498806ead0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f499c13e820 0x7f499c133120 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] conn(0x7f499c0b0ca0 0x7f499c140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:43.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 --2- 172.21.15.47:0/3046948969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f499c071be0 0x7f499c1d9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T07:45:43.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 >> 172.21.15.47:0/3046948969 conn(0x7f499c0b73d0 msgr2=0x7f499c138560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T07:45:43.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 shutdown_connections 2024-03-30T07:45:43.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 -- 172.21.15.47:0/3046948969 wait complete. 2024-03-30T07:45:43.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-03-30T07:45:43.279+0000 7f49a456e640 1 librados: shutdown 2024-03-30T07:45:43.294 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-30T07:45:43.294 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-30T07:45:43.295 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-30T07:45:43.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:45:43.311 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:45:43.297+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35180) UID: 0 2024-03-30T07:45:43.311 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-03-30T07:45:43.298+0000 b47e640 -1 osd.0 994 *** Got signal Terminated *** 2024-03-30T07:45:52.572 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:20.383 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.575 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:20.385 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.593 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:20.404 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.624 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:20.435 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.628 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:20.439 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.646 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:20.456 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.653 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:20.430 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.654 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:20.431 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:45:52.671 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:20.448 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.517 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:30.328 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.521 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:30.332 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.568 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:30.379 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.571 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:30.382 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.576 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:30.387 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.596 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:30.373 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.597 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:30.374 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.612 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:30.389 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:02.619 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:30.430 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.437 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:35.248 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.488 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:35.298 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.504 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:35.315 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.538 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:35.349 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.541 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:35.318 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.541 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:35.352 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.542 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:35.319 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.557 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:35.367 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:07.557 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:35.334 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:19.902 INFO:tasks.ceph.osd.0:Stopped 2024-03-30T07:46:19.902 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T07:46:19.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:46:19.916 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:46:19.905+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35186) UID: 0 2024-03-30T07:46:19.916 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-03-30T07:46:19.907+0000 b47e640 -1 osd.1 1002 *** Got signal Terminated *** 2024-03-30T07:46:22.606 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:50.416 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.609 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:50.419 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.613 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:50.390 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.615 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:50.426 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.616 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:50.427 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.665 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:50.442 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.682 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:50.459 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.707 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:50.518 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:22.708 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:50.518 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:23.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-30T07:46:26.003 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T07:46:26.004 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-30T07:46:26.004 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:46:26.027 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:46:26.008+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35197) UID: 0 2024-03-30T07:46:26.028 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-03-30T07:46:26.010+0000 b47e640 -1 osd.2 1006 *** Got signal Terminated *** 2024-03-30T07:46:27.675 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:55.485 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.711 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:55.522 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.725 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:55.502 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.728 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:55.505 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.731 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:30:55.542 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.744 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:30:55.522 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.786 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:55.596 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.788 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:55.598 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:27.801 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:30:55.612 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:29.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-30T07:46:29.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-30T07:46:32.107 INFO:tasks.ceph.osd.2:Stopped 2024-03-30T07:46:32.107 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T07:46:32.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:46:32.121 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:46:32.110+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35196) UID: 0 2024-03-30T07:46:32.124 INFO:tasks.ceph.osd.3.smithi047.stderr:2024-03-30T07:46:32.111+0000 b47e640 -1 osd.3 1009 *** Got signal Terminated *** 2024-03-30T07:46:35.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-30T07:46:35.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-30T07:46:38.209 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T07:46:38.210 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T07:46:38.210 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:46:38.223 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:46:38.212+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35304) UID: 0 2024-03-30T07:46:38.223 INFO:tasks.ceph.osd.4.smithi052.stderr:2024-03-30T07:46:38.213+0000 b47e640 -1 osd.4 1012 *** Got signal Terminated *** 2024-03-30T07:46:41.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-30T07:46:41.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-30T07:46:42.658 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:10.468 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.660 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:10.471 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.679 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:10.490 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.707 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:10.517 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.711 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:10.522 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.711 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:10.488 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.713 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:10.490 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.730 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:10.541 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:42.733 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:10.510 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.369 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:11.180 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.370 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:11.181 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.379 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:11.189 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.410 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:11.187 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.411 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:11.188 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.417 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:11.228 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.418 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:11.229 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.419 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:11.196 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:43.428 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:11.239 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:46:44.311 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T07:46:44.311 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T07:46:44.312 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:46:44.328 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:46:44.317+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35307) UID: 0 2024-03-30T07:46:44.328 INFO:tasks.ceph.osd.5.smithi052.stderr:2024-03-30T07:46:44.319+0000 b47e640 -1 osd.5 1014 *** Got signal Terminated *** 2024-03-30T07:46:47.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-30T07:47:02.720 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:30.530 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.721 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:30.532 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.740 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:30.551 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.767 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:30.544 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.768 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:30.579 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.771 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:30.581 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.771 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:30.548 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.789 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:30.600 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:02.790 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:30.567 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.442 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:31.252 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.443 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:31.253 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.451 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:31.262 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.464 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:31.242 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.466 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:31.243 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.474 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:31.251 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.489 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:31.300 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.491 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:31.301 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:03.499 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:31.310 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.109 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:33.920 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.112 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:33.923 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.132 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:33.943 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.161 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:33.938 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.164 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:33.941 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.178 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:33.955 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.179 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:33.989 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.180 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:33.991 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:06.194 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:34.004 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.784 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:50.594 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.785 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:50.596 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.800 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:50.610 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.834 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:50.611 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.835 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:50.646 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.835 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:50.613 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.836 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:50.647 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.849 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:50.626 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:22.850 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:50.661 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.794 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:55.604 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.798 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:55.607 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.811 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:31:55.621 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.843 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:55.620 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.846 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:55.623 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.847 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:55.658 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.849 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:55.660 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.857 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:31:55.633 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:27.860 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:31:55.671 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.839 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:10.649 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.843 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:10.654 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.860 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:10.671 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.891 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:10.668 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.894 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:10.671 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.898 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:10.708 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.899 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:10.710 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.909 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:10.686 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:42.913 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:10.724 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.810 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:15.620 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.859 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:15.636 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.860 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:15.637 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.863 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:15.673 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.863 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:15.674 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.864 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:15.675 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.871 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:15.649 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.879 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:15.690 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:47:47.884 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:15.695 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.917 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:30.728 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.923 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:30.733 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.940 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:30.751 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.972 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:30.783 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.972 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:30.749 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.974 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:30.751 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.975 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:30.785 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.990 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:30.767 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:02.991 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:30.802 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.936 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:35.746 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.940 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:35.751 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.954 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:35.764 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.987 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:35.765 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.990 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:35.801 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.991 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:35.768 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:07.993 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:35.804 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:08.002 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:35.779 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:08.007 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:35.818 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:22.992 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:50.803 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:22.994 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:50.805 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.010 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:50.821 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.042 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:50.853 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.043 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:50.820 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.044 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:50.854 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.045 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:50.822 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.060 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:50.837 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:23.060 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:50.871 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.003 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:55.813 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.007 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:55.817 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.024 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:32:55.835 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.058 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:55.869 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.058 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:55.835 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.061 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:55.872 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.065 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:55.842 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.073 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:32:55.884 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:28.076 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:32:55.853 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:31.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-30T07:48:34.140 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T07:48:34.140 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T07:48:34.140 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:48:34.155 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:48:34.143+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35309) UID: 0 2024-03-30T07:48:34.157 INFO:tasks.ceph.osd.6.smithi052.stderr:2024-03-30T07:48:34.144+0000 b47e640 -1 osd.6 1016 *** Got signal Terminated *** 2024-03-30T07:48:37.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-30T07:48:43.051 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:10.861 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.056 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:10.866 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.078 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:10.888 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.105 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:10.882 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.108 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:10.885 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.111 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:10.922 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.115 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:10.925 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.124 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:10.902 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:43.134 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:10.943 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.238 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:16.048 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.240 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:16.051 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.254 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:16.064 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.284 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:16.094 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.286 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:16.097 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.287 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:16.064 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.290 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:16.067 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.298 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:16.109 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:48:48.302 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:16.079 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.147 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:30.957 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.149 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:30.960 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.166 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:30.976 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.196 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:30.973 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.198 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:30.975 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.200 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:31.010 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.202 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:31.013 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.215 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:30.992 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:03.219 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:31.030 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.143 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:35.953 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.197 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:36.007 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.214 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:36.024 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.243 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:36.020 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.245 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:36.022 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.246 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:36.057 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.251 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:36.062 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.260 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:36.037 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:08.265 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:36.076 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.220 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:51.030 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.224 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:51.035 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.242 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:51.053 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.271 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:51.082 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.274 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:51.051 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.276 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:51.053 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.276 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:51.087 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.293 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:51.103 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:23.297 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:51.074 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.232 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:56.043 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.237 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:56.048 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.250 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:33:56.061 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.287 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:56.097 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.288 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:56.065 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.288 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:56.099 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.289 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:56.066 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.301 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:33:56.112 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:28.305 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:33:56.082 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.280 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:11.091 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.285 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:11.095 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.309 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:11.120 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.334 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:11.111 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.338 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:11.115 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.341 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:11.152 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.344 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:11.155 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.355 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:11.132 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:43.364 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:11.174 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.250 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:16.060 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.302 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:16.112 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.314 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:16.125 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.347 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:16.124 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.350 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:16.160 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.352 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:16.162 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.353 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:16.130 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.364 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:16.141 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:48.367 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:16.178 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:49:56.388 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:49:56.384+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:49:57.416 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:49:57.414+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:49:58.434 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:49:58.432+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:49:59.435 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:49:59.433+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:00.476 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:00.474+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:01.444 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:01.441+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:02.414 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:02.412+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:03.345 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:31.155 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.349 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:31.159 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.368 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:31.178 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.396 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:31.173 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.396 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:31.207 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.401 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:31.212 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.403 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:31.180 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.417 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:31.228 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.426 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:31.203 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:03.432 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:03.429+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:04.479 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:04.475+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:05.508 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:05.506+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:06.470 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:06.467+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:07.507 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:07.503+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:08.359 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:36.170 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.362 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:36.173 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.372 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:36.183 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.408 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:36.219 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.411 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:36.221 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.411 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:36.188 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.413 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:36.190 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.422 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:36.233 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.426 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:36.203 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:08.471 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:08.468+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:09.507 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:09.504+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:10.498 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:10.495+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:11.486 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:11.482+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:12.492 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:12.489+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:13.539 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:13.536+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:14.571 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:14.568+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:15.588 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:15.584+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:16.558 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:16.555+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:17.547 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:17.544+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:18.529 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:46.340 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.531 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:46.342 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.550 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:46.360 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.576 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:18.573+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:18.583 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:46.360 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.587 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:46.397 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.587 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:46.364 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.588 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:46.399 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.605 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:46.415 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:18.611 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:46.388 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:19.609 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:19.606+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:20.586 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:20.582+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:21.549 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:21.546+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:22.555 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:22.552+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:23.410 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:51.221 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.413 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:51.224 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.426 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:34:51.237 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.458 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:51.235 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.461 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:51.238 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.465 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:51.275 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.466 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:51.277 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.472 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:34:51.249 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.478 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:34:51.289 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:23.543 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:23.540+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:24.557 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:24.554+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:25.519 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:25.517+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:26.529 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:26.526+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:27.480 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:27.477+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:28.509 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:28.505+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:29.531 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:29.528+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:30.523 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:30.520+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:31.516 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:31.513+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:32.558 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:32.555+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:33.575 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:33.572+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:34.543 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:34.540+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:35.518 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:35.514+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:36.496 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:36.493+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:37.489 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:37.486+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:38.498 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:38.495+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:39.547 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:39.543+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:40.517 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:40.514+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:41.558 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:41.555+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:42.532 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:42.528+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:43.471 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:11.282 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.476 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:11.287 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.497 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:11.307 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.525 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:11.302 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.526 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:11.304 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.528 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:11.338 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.530 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:11.341 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.541 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:43.538+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:43.545 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:11.322 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:43.549 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:11.360 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:44.545 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:44.542+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:45.552 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:45.549+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:46.594 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:46.591+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:47.560 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:47.556+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:48.441 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:16.251 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.494 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:16.304 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.506 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:16.316 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.543 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:16.353 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.544 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:16.321 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.547 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:16.357 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.547 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:16.324 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.558 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:16.335 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.559 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:16.370 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:50:48.570 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:48.568+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:49.524 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:49.520+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:50.570 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:50.566+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:51.526 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:51.523+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:52.490 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:52.485+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:53.473 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:53.469+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:54.518 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:54.515+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:55.547 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:55.544+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:56.530 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:56.527+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:57.487 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:57.484+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:58.486 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:58.482+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:50:59.452 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:50:59.449+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:00.423 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:00.420+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:01.388 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:01.384+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:02.376 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:02.373+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:03.382 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:03.379+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:03.542 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:31.353 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.544 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:31.355 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.560 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:31.371 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.589 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:31.367 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.593 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:31.370 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.599 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:31.409 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.600 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:31.411 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.610 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:31.387 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:03.616 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:31.427 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:04.408 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:04.405+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:05.436 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:05.433+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:06.473 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:06.469+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:07.432 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:07.429+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:08.399 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:08.397+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:08.605 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:36.416 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.636 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:36.447 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.650 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:36.460 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.651 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:36.428 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.653 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:36.430 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.659 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:36.470 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.688 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:36.499 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.692 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:36.469 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:08.701 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:36.512 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:09.427 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:09.425+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:10.400 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:10.398+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:11.365 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:11.362+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:12.346 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:12.344+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:13.366 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:13.364+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:14.357 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:14.355+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:15.319 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:15.317+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:16.310 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:16.307+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:17.331 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:17.328+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:18.358 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:18.356+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:19.403 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:19.400+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:20.376 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:20.372+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:21.407 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:21.404+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:22.438 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:22.435+0000 12a60640 -1 osd.7 1016 heartbeat_check: no reply from 172.21.15.52:6808 osd.6 since back 2024-03-30T07:48:32.216366+0000 front 2024-03-30T07:48:32.217488+0000 (oldest deadline 2024-03-30T07:49:56.313826+0000) 2024-03-30T07:51:28.689 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:56.499 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.693 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:56.503 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.728 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:35:56.539 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.742 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:56.519 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.743 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:56.520 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.751 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:56.562 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.756 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:56.567 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.764 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:35:56.541 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:28.790 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:35:56.601 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.736 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:06.547 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.739 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:06.550 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.763 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:06.573 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.786 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:06.563 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.788 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:06.565 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.796 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:06.606 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.798 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:06.609 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.801 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:06.578 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:38.812 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:06.622 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:43.285 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T07:51:43.444 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T07:51:43.444 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:51:43.458 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:43.447+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35311) UID: 0 2024-03-30T07:51:43.459 INFO:tasks.ceph.osd.7.smithi052.stderr:2024-03-30T07:51:43.449+0000 b47e640 -1 osd.7 1016 *** Got signal Terminated *** 2024-03-30T07:51:45.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-30T07:51:48.769 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:16.579 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.772 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:16.583 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.788 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:16.599 34885== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.819 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:16.630 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.821 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:16.598 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.822 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:16.633 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.824 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:16.601 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.836 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:16.646 34887== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:48.837 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:16.614 34693== Warning: unimplemented fcntl command: 1036 2024-03-30T07:51:49.546 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T07:51:49.546 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-30T07:51:49.546 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-30T07:51:49.546 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:51:49.569 INFO:tasks.ceph.mgr.y:Stopped 2024-03-30T07:51:49.570 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-30T07:51:49.570 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:51:49.596 INFO:tasks.ceph.mgr.x:Stopped 2024-03-30T07:51:49.596 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-30T07:51:49.597 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-30T07:51:49.597 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:51:49.625 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-03-30T07:51:49.612+0000 9f4f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34858) UID: 0 2024-03-30T07:51:49.625 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-03-30T07:51:49.613+0000 9f4f640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T07:51:50.199 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:18.009 34885== 2024-03-30T07:51:50.199 INFO:tasks.ceph.mon.a.smithi047.stderr:==00:00:36:18.009 34885== Exit program on first error (--exit-on-first-error=yes) 2024-03-30T07:51:50.218 INFO:tasks.ceph.mon.a.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-30T07:51:50.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-30T07:51:50.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-30T07:51:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-30T07:51:55.698 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-30T07:51:55.699 INFO:tasks.ceph.mon.a:Stopped 2024-03-30T07:51:55.700 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-30T07:51:55.700 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:51:55.716 INFO:tasks.ceph.mon.c.smithi047.stderr:2024-03-30T07:51:55.703+0000 9f4f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34860) UID: 0 2024-03-30T07:51:55.717 INFO:tasks.ceph.mon.c.smithi047.stderr:2024-03-30T07:51:55.704+0000 9f4f640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-30T07:51:55.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-30T07:51:56.443 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:24.253 34887== 2024-03-30T07:51:56.443 INFO:tasks.ceph.mon.c.smithi047.stderr:==00:00:36:24.253 34887== Exit program on first error (--exit-on-first-error=yes) 2024-03-30T07:51:56.463 INFO:tasks.ceph.mon.c.smithi047.stderr:daemon-helper: command failed with exit status 42 2024-03-30T07:52:01.012 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-30T07:52:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-30T07:52:01.803 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi047 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-30T07:52:01.803 INFO:tasks.ceph.mon.c:Stopped 2024-03-30T07:52:01.803 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-30T07:52:01.803 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T07:52:01.821 INFO:tasks.ceph.mon.b.smithi052.stderr:2024-03-30T07:52:01.806+0000 9f4f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34688) UID: 0 2024-03-30T07:52:01.822 INFO:tasks.ceph.mon.b.smithi052.stderr:2024-03-30T07:52:01.807+0000 9f4f640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-30T07:52:02.627 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:30.404 34693== 2024-03-30T07:52:02.627 INFO:tasks.ceph.mon.b.smithi052.stderr:==00:00:36:30.404 34693== Exit program on first error (--exit-on-first-error=yes) 2024-03-30T07:52:02.644 INFO:tasks.ceph.mon.b.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-30T07:52:06.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-30T07:52:06.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-30T07:52:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-30T07:52:07.906 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-30T07:52:07.906 INFO:tasks.ceph.mon.b:Stopped 2024-03-30T07:52:07.906 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-30T07:52:07.907 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-30T07:52:07.948 INFO:teuthology.orchestra.run.smithi047.stdout:2024-03-30T07:22:26.894227+0000 osd.4 (osd.4) 16 : cluster [WRN] osd.4 ep: 195 scrubber::ReplicaReservations pg[11.9]: timeout on replica reservations (since 2024-03-30 07:22:21) 2024-03-30T07:52:07.948 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-30T07:52:07.948 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-30T07:52:08.022 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-30T07:52:08.097 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-30T07:52:08.170 INFO:teuthology.orchestra.run.smithi047.stdout:2024-03-30T07:22:26.894227+0000 osd.4 (osd.4) 16 : cluster [WRN] osd.4 ep: 195 scrubber::ReplicaReservations pg[11.9]: timeout on replica reservations (since 2024-03-30 07:22:21) 2024-03-30T07:52:08.170 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:52:08.171 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-30T07:52:08.321 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:52:08.322 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-30T07:52:08.501 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:52:08.502 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-30T07:52:08.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:52:08.568 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-30T07:52:08.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:52:08.627 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-30T07:52:08.877 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:52:08.877 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-30T07:52:08.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:52:08.942 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-30T07:52:09.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:52:09.016 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-30T07:52:09.073 INFO:tasks.ceph:Archiving mon data... 2024-03-30T07:52:09.075 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/data/mon.a.tgz 2024-03-30T07:52:09.075 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-30T07:52:09.091 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.WkqG0uvlBd 2024-03-30T07:52:09.092 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WkqG0uvlBd 2024-03-30T07:52:09.265 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.WkqG0uvlBd 2024-03-30T07:52:09.351 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.WkqG0uvlBd is 502KB 2024-03-30T07:52:09.451 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.WkqG0uvlBd 2024-03-30T07:52:09.465 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/data/mon.c.tgz 2024-03-30T07:52:09.465 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-03-30T07:52:09.519 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.S6ZBU9LO11 2024-03-30T07:52:09.519 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.S6ZBU9LO11 2024-03-30T07:52:09.678 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.S6ZBU9LO11 2024-03-30T07:52:09.762 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.S6ZBU9LO11 is 502KB 2024-03-30T07:52:09.859 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.S6ZBU9LO11 2024-03-30T07:52:09.873 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/data/mon.b.tgz 2024-03-30T07:52:09.873 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-30T07:52:09.890 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.sUtJRCxA8J 2024-03-30T07:52:09.891 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.sUtJRCxA8J 2024-03-30T07:52:10.071 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.sUtJRCxA8J 2024-03-30T07:52:10.158 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.sUtJRCxA8J is 502KB 2024-03-30T07:52:10.266 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.sUtJRCxA8J 2024-03-30T07:52:10.281 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-30T07:52:10.281 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T07:52:10.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T07:52:10.347 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-30T07:52:10.348 DEBUG:teuthology.orchestra.run.smithi047:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-30T07:52:10.351 DEBUG:teuthology.orchestra.run.smithi052:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-30T07:52:10.391 INFO:teuthology.orchestra.run.smithi047.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2024-03-30T07:52:10.391 INFO:teuthology.orchestra.run.smithi047.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2024-03-30T07:52:10.391 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2024-03-30T07:52:10.391 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2024-03-30T07:52:10.391 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2024-03-30T07:52:10.391 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2024-03-30T07:52:10.427 INFO:teuthology.orchestra.run.smithi052.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2024-03-30T07:52:10.428 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2024-03-30T07:52:10.428 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2024-03-30T07:52:10.428 INFO:tasks.ceph:Archiving crash dumps... 2024-03-30T07:52:10.430 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi047/crash 2024-03-30T07:52:10.430 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T07:52:10.457 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi052/crash 2024-03-30T07:52:10.458 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T07:52:10.494 INFO:tasks.ceph:Compressing logs... 2024-03-30T07:52:10.494 DEBUG:teuthology.orchestra.run.smithi047:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T07:52:10.501 DEBUG:teuthology.orchestra.run.smithi052:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T07:52:11.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-30T07:52:35.804 INFO:tasks.ceph:Archiving logs... 2024-03-30T07:52:35.804 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi047/log 2024-03-30T07:52:35.805 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T07:52:38.964 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi052/log 2024-03-30T07:52:38.965 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T07:52:41.612 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-03-30T07:52:41.613 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T07:52:41.623 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_55c4b44e7552b7e7285ac1c9bbe0a46dff3edb83/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-03-30T07:52:41.624 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T07:52:41.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T07:52:41.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T07:52:41.710 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T07:52:41.720 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T07:52:41.720 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T07:52:41.722 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T07:52:41.736 INFO:teuthology.orchestra.run.smithi047.stderr:bash: line 1: ntpq: command not found 2024-03-30T07:52:41.739 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-30T07:52:41.741 INFO:teuthology.orchestra.run.smithi047.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T07:52:41.741 INFO:teuthology.orchestra.run.smithi047.stdout:=============================================================================== 2024-03-30T07:52:41.742 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -16us[ -38us] +/- 68ms 2024-03-30T07:52:41.742 INFO:teuthology.orchestra.run.smithi047.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 +23us[+1149ns] +/- 79ms 2024-03-30T07:52:41.742 INFO:teuthology.orchestra.run.smithi047.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 100 +19us[ -30us] +/- 90ms 2024-03-30T07:52:41.742 INFO:teuthology.orchestra.run.smithi047.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 102 +14us[ +26us] +/- 67ms 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 7 377 96 +14us[ +26us] +/- 78ms 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +12us[ +12us] +/- 91ms 2024-03-30T07:52:41.744 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T07:52:41.745 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T07:52:41.754 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T07:52:41.755 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T07:52:41.763 DEBUG:teuthology.orchestra.run.smithi047:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T07:52:41.856 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T07:52:41.929 DEBUG:teuthology.orchestra.run.smithi047:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T07:52:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T07:52:41.962 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T07:52:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T07:52:41.994 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T07:52:42.009 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T07:52:42.022 INFO:teuthology.task.internal:Duration was 3332.345771 seconds 2024-03-30T07:52:42.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T07:52:42.038 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T07:52:42.038 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T07:52:42.042 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T07:52:42.084 INFO:teuthology.orchestra.run.smithi047.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T07:52:42.089 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T07:52:42.301 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T07:52:42.301 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-03-30T07:52:42.301 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T07:52:42.321 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-03-30T07:52:42.321 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T07:52:42.353 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T07:52:42.353 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T07:52:42.364 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T07:52:42.422 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T07:52:42.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T07:52:42.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T07:52:42.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T07:52:42.577 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T07:52:42.577 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T07:52:42.604 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T07:52:42.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T07:52:42.645 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T07:52:42.648 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T07:52:42.670 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-03-30T07:52:42.697 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-03-30T07:52:42.718 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T07:52:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T07:52:42.742 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T07:52:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T07:52:42.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T07:52:42.785 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T07:52:42.786 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi047 2024-03-30T07:52:42.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T07:52:42.823 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630/remote/smithi052 2024-03-30T07:52:42.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T07:52:42.860 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T07:52:42.860 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T07:52:42.863 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T07:52:42.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T07:52:42.920 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T07:52:42.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T07:52:42.936 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T07:52:42.936 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T07:52:42.939 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T07:52:42.950 INFO:teuthology.orchestra.run.smithi047.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 07:52 /home/ubuntu/cephtest 2024-03-30T07:52:42.963 INFO:teuthology.orchestra.run.smithi052.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 07:52 /home/ubuntu/cephtest 2024-03-30T07:52:42.964 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T07:52:42.974 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T07:52:42.985 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2024-03-30T07:52:42.985 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T07:52:43.011 DEBUG:teuthology.nuke:shortname: smithi047 2024-03-30T07:52:43.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T07:52:43.037 DEBUG:teuthology.nuke:shortname: smithi052 2024-03-30T07:52:43.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T07:52:43.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 06:48:31.693517', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCp1NYDh5mND57I3ETURfiQ7fqoDP3o6d3ZlghAYPMrkU0/MSYTdfBhjOBfSrbPKToN6CgqW5VCkIWDPzd1CX38='} 2024-03-30T07:52:43.060 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-29_19:10:12-rados-wip-yuri11-testing-2024-03-28-0753-reef-distro-default-smithi/7629630', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 06:48:31.692125', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMfXVMcbopoc/81qCl9RxIlD5evhICEsqNOexv19l/QmBJ4aBrFJZwxWdExXnqpSLRdVORGZlHFF2f8UAo1Zr0='} 2024-03-30T07:52:43.089 INFO:teuthology.orchestra.console.smithi047:Power off 2024-03-30T07:52:43.089 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T07:52:43.105 INFO:teuthology.orchestra.console.smithi052:Power off 2024-03-30T07:52:43.106 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T07:52:43.117 DEBUG:teuthology.orchestra.console.smithi047:power off output: Chassis Power Control: Down/Off 2024-03-30T07:52:43.117 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:43.130 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-03-30T07:52:43.131 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:43.142 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-30T07:52:43.156 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-30T07:52:47.143 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:47.157 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:47.255 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is on 2024-03-30T07:52:47.266 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-30T07:52:51.257 DEBUG:teuthology.orchestra.console.smithi047:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:51.268 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T07:52:51.367 DEBUG:teuthology.orchestra.console.smithi047:check power output: Chassis Power is off 2024-03-30T07:52:51.381 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-03-30T07:52:51.468 INFO:teuthology.orchestra.console.smithi047:Power off completed 2024-03-30T07:52:51.482 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-03-30T07:52:51.586 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3332.3457708358765 failure_reason: '"2024-03-30T07:22:26.894227+0000 osd.4 (osd.4) 16 : cluster [WRN] osd.4 ep: 195 scrubber::ReplicaReservations pg[11.9]: timeout on replica reservations (since 2024-03-30 07:22:21)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-30T07:52:51.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T07:52:51.672 INFO:teuthology.run:FAIL